分类目录归档：技术笔记

修复 Joplin on KDE 菜单栏显示问题

发表评论

在 KDE 桌面下默认使用全局菜单显示应用程序的菜单栏，但是唯独 Joplin 无法显示。

最后在这里找到了解决方案，下面简单记录：

sudo vim /usr/share/applications/joplin-desktop.desktop 
--Exec=/usr/bin/joplin-desktop
++Exec=/usr/bin/joplin-desktop --gtk-version=3 --ozone-platform=x11

启动程序增加这两个参数即可解决。

Refs

我的博客即将同步至腾讯云开发者社区，邀请大家一同入驻：https://cloud.tencent.com/developer/support-plan?invite_code=21yjpwt8mhhc0

Copy Fail：Linux 内核 2017 年至今的高危漏洞（附临时缓解方案） | CVE-2026-31431

发表评论

最近爆出一个 Linux 内核存在近10年的漏洞，随便找了一台最近在用的机器试了一下，直接成功提权：

~$ whoami
songtianlun
~$ python3 test.py 
# whoami
root

临时解决方案如下：

由于 SSH/OpenSSL 等安全基建库几乎都在使用自行维护的用户态加密库, 所以 AF_ALG 可以直接禁用, 为临时缓解措施 (仅供参考):

rmmod algif_aead 2>/dev/null || true
echo "install algif_aead /bin/false" > /etc/modprobe.d/disable-algif.conf

再尝试就不行了：

~$ python3 test.py Traceback (most recent call last):
  File "/home/songtianlun/test.py", line 9, in 
<module>
    while i<len(e):c(f,i,e[i:i+4]);i+=4
  File "/home/songtianlun/test.py", line 5, in c
    a=s.socket(38,5,0);a.bind(("aead","authencesn(hmac(sha256),cbc(aes))"));h=279;v=a.setsockopt;v(h,1,d('0800010000000010'+'0'*64));v(h,5,None,4);u,_=a.accept();o=t+4;i=d('00');u.sendmsg([b"A"*4+c],[(h,3,i*4),(h,2,b'\x10'+i*19),(h,4,b'\x08'+i*3),],32768);r,w=g.pipe();n=g.splice;n(f,w,o,offset_src=0);n(r,u.fileno(),o)
FileNotFoundError: [Errno 2] No such file or directory

影响面甚广，赶快检查一下自己的服务器。

Refs

Hermes Agent — 在 K3s / K8s 中运行指南

发表评论

本文基于官方 Docker 文档，将 Hermes Agent 迁移到 Kubernetes / K3s 环境，使用 StatefulSet 管理持久化工作负载。

1. 前置准备

  • K3s 或 K8s 集群已就绪（本文以 K3s 为例）
  • 节点上已有 containerd（K3s 默认内置）
  • 推荐安装 nerdctl 作为容器管理工具（参考：在 K3s 节点上安装并使用 nerdctl）
  • 镜像：nousresearch/hermes-agent:latest

2. 初始化配置（持久化数据目录）

在首次运行前，需要先执行一次 Setup Wizard，将 API Keys 等配置写入宿主机目录，再挂载进容器使用。

这里建议使用 nerdctl 运行，其他的方法需自行探索。

# 在目标节点上创建数据目录
mkdir -p /var/lib/hermes-data

# 使用 nerdctl 运行一次性 setup 容器（交互模式）
sudo nerdctl run -it --rm \
  -v /var/lib/hermes-data:/opt/data \
  nousresearch/hermes-agent:latest setup

配置完成后的数据目录结构

/var/lib/hermes-data/
├── .env            # API Keys 与密钥
├── config.yaml     # 主配置文件
├── SOUL.md         # Agent 人格 / 身份设定
├── sessions/       # 会话历史
├── memories/       # 持久记忆
├── skills/         # 已安装的技能
├── cron/           # 定时任务定义
├── hooks/          # 事件钩子
├── logs/           # 运行日志
└── skins/          # 自定义 CLI 皮肤

3. 部署 Gateway 后台服务（StatefulSet）

这里直接给出参考 yaml，按需调整：

---
apiVersion: v1
kind: Service
metadata:
  name: gateway
  namespace: hermes
spec:
  selector:
    app: gateway
  ports:
    - name: api
      port: 8642
      targetPort: 8642
  type: ClusterIP
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: data
  namespace: hermes
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Gi
  storageClassName: nfs-hhus3
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: gateway
  namespace: hermes
spec:
  serviceName: gateway
  replicas: 1
  selector:
    matchLabels:
      app: gateway
  template:
    metadata:
      labels:
        app: gateway
    spec:
      nodeSelector:
        hosthatch/zone: lax
      containers:
        - name: gateway
          image: nousresearch/hermes-agent:latest
          args: ["gateway", "run"]
          ports:
            - containerPort: 8642
          env:
            - name: TZ
              value: "Asia/Shanghai"
          volumeMounts:
            - name: hermes-data
              mountPath: /opt/data
          resources:
            requests:
              memory: "1Gi"
              cpu: "500m"
            limits:
              memory: "4Gi"
              cpu: "2"
      volumes:
        - name: hermes-data
          persistentVolumeClaim:
            claimName: data

4. 部署 Dashboard 仪表盘（StatefulSet）

直接给出参考 yaml，按需调整：

apiVersion: v1
kind: Service
metadata:
  name: dashboard
  namespace: hermes
spec:
  selector:
    app: dashboard
  ports:
    - name: web
      port: 9119
      targetPort: 9119
  type: ClusterIP   # 按需改为 NodePort / LoadBalancer
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dashboard
  namespace: hermes
spec:
  serviceName: dashboard
  replicas: 1
  selector:
    matchLabels:
      app: dashboard
  template:
    metadata:
      labels:
        app: dashboard
    spec:
      nodeSelector:
        hosthatch/zone: lax
      containers:
        - name: dashboard
          image: nousresearch/hermes-agent:latest
          args: ["dashboard"]
          #args: ["dashboard", "--host", "0.0.0.0", "--insecure"]
          ports:
            - containerPort: 9119
          env:
            # 指向 Gateway Service 的 ClusterIP DNS 名称
            - name: GATEWAY_HEALTH_URL
              value: "http://gateway.hermes.svc.cluster.local:8642"
            - name: GATEWAY_HEALTH_TIMEOUT
              value: "3"
          volumeMounts:
            - name: hermes-data
              mountPath: /opt/data
              readOnly: true    # Dashboard 只读数据目录
          resources:
            requests:
              memory: "256Mi"
              cpu: "100m"
            limits:
              memory: "512Mi"
              cpu: "500m"

      volumes:
        - name: hermes-data
          persistentVolumeClaim:
            claimName: data

可以使用 port-forward 安全的访问仪表盘，不建议对外暴露：

kubectl port-forward -n hermes svc/hermes-dashboard 9119:9119
# 浏览器访问 http://localhost:9119

5. 运行交互式 CLI 聊天

在已部署并配置好的数据目录基础上，可随时进行交互式聊天，使用 kubectl exec 进入 Gateway 容器

kubectl exec -it -n hermes gateway-0 -- /opt/hermes/.venv/bin/hermes

Refs

在 K3s 节点上安装并使用 nerdctl

发表评论

适用场景：K3s 默认不附带 nerdctl，但其内置的 containerd 与 nerdctl 完全兼容。本教程讲解如何在 K3s 节点上以最小代价安装 nerdctl，并正确指向 K3s 的 containerd socket，无需重复安装 containerd 或 CNI。

一、背景与原理

工具 说明
ctr containerd 内置调试工具，与 Docker CLI 不兼容，功能有限
crictl CRI 调试工具，K3s 自带，面向 Kubernetes 运维
nerdctl Docker 兼容 CLI，支持 run/build/compose，推荐日常使用

K3s 的 containerd socket 路径为 /run/k3s/containerd/containerd.sock，而非标准路径 /run/containerd/containerd.sock。只需在配置中指向该路径，nerdctl 即可接管 K3s 容器管理。

K3s 已自带 CNI 插件（flannel/calico 等），查看 K3s 节点已有的 Pod 和镜像无需额外 CNI。若需要 nerdctl run 启动独立容器并连接网络，则需要补充安装 CNI 插件（见第四节）。

二、安装 nerdctl（仅二进制）

K3s 节点已有 containerd，只需下载 nerdctl 的精简包（不含 containerd/CNI，体积小）。

2.1 下载二进制

# 查询最新版本（或手动前往 https://github.com/containerd/nerdctl/releases 查看）
NERDCTL_VERSION=$(curl -s https://api.github.com/repos/containerd/nerdctl/releases/latest \
  | grep tag_name | cut -d '"' -f4 | tr -d 'v')

echo "最新版本: ${NERDCTL_VERSION}"

# 下载精简包（仅 nerdctl 二进制）
curl -LO "https://github.com/containerd/nerdctl/releases/download/v${NERDCTL_VERSION}/nerdctl-${NERDCTL_VERSION}-linux-amd64.tar.gz"

ARM64 节点（如树莓派、ARM 服务器）将 amd64 替换为 arm64：

curl -LO "https://github.com/containerd/nerdctl/releases/download/v${NERDCTL_VERSION}/nerdctl-${NERDCTL_VERSION}-linux-arm64.tar.gz"

2.2 解压并安装

# 解压到 /usr/local/bin
sudo tar Cxzvf /usr/local/bin nerdctl-${NERDCTL_VERSION}-linux-amd64.tar.gz nerdctl

# 验证安装
nerdctl --version

三、配置 nerdctl 指向 K3s containerd

nerdctl 默认连接 /run/containerd/containerd.sock，在 K3s 节点上需要修改为 K3s 专用路径。

3.1 创建配置文件

sudo mkdir -p /etc/nerdctl

sudo tee /etc/nerdctl/nerdctl.toml > /dev/null <<EOF
# nerdctl 全局配置，适配 K3s 节点
address        = "/run/k3s/containerd/containerd.sock"
namespace      = "k8s.io"
EOF

说明：

  • address：K3s containerd 的 socket 路径
  • namespace：K3s 所有容器和镜像均存储在 k8s.io 命名空间下

3.2 验证连接

# 列出 K3s 命名空间下的所有容器（等同于 kubectl get pods 的容器视角）
sudo nerdctl ps -a

# 列出镜像
sudo nerdctl images

如果能看到 K3s 系统 Pod（如 coredns、traefik 等），说明配置成功。

四、安装 CNI 插件（按需，用于 nerdctl run）

如果只需要查看 K3s 已有容器和镜像，可跳过此节。

只有当你需要用 nerdctl run 启动独立容器（即非 Kubernetes 管理的容器）时，才需要 CNI 插件。K3s 自带的 CNI 仅供 Kubernetes 使用，nerdctl 的独立容器网络需要单独配置。

4.1 下载官方 CNI 插件

CNI_VERSION=$(curl -s https://api.github.com/repos/containernetworking/plugins/releases/latest \
  | grep tag_name | cut -d '"' -f4)

curl -LO "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz"

# 安装到标准路径
sudo mkdir -p /opt/cni/bin
sudo tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-${CNI_VERSION}.tgz

4.2 创建默认网络配置

sudo mkdir -p /etc/cni/net.d

sudo tee /etc/cni/net.d/10-nerdctl-bridge.conflist > /dev/null <<EOF
{
  "cniVersion": "1.0.0",
  "name": "nerdctl-bridge",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "nerdctl0",
      "isGateway": true,
      "ipMasq": true,
      "ipam": {
        "type": "host-local",
        "ranges": [
          [{"subnet": "10.88.0.0/16"}]
        ],
        "routes": [{"dst": "0.0.0.0/0"}]
      }
    },
    {
      "type": "portmap",
      "capabilities": {"portMappings": true}
    },
    {
      "type": "firewall"
    }
  ]
}
EOF

注意：此桥接网络（10.88.0.0/16）仅供 nerdctl 管理的独立容器使用，不会影响 K3s 自身网络。

4.3 验证独立容器运行

# 注意：启动独立容器时需使用默认命名空间（不加 --namespace k8s.io）
# 或在 nerdctl.toml 中临时切换，推荐直接在命令行覆盖：
sudo nerdctl --namespace default run -d --name test-nginx -p 8080:80 nginx:alpine

# 确认运行
sudo nerdctl --namespace default ps
curl http://localhost:8080

五、常用命令速查

所有命令均需 sudo（或将当前用户加入 containerd 相关权限组）。

查看 K3s 容器和镜像

# 列出所有容器（K3s 管理）
sudo nerdctl ps -a

# 列出镜像
sudo nerdctl images

# 查看容器日志
sudo nerdctl logs <容器ID或名称>

# 进入容器终端
sudo nerdctl exec -it <容器ID或名称> sh

镜像管理

# 拉取镜像（拉取后可直接被 K3s Pod 使用）
sudo nerdctl pull nginx:alpine

# 查看镜像详情
sudo nerdctl inspect <镜像ID>

# 删除镜像
sudo nerdctl rmi <镜像ID>

# 从 tar 包导入镜像（常用于离线环境）
sudo nerdctl load < image.tar

# 导出镜像为 tar 包
sudo nerdctl save nginx:alpine -o nginx.tar

构建镜像（需安装 BuildKit，见第六节）

sudo nerdctl build -t myapp:v1 /path/to/dockerfile-dir

配合 kubectl 使用本地镜像

# 构建并打 tag 到 k8s.io 命名空间
sudo nerdctl --namespace k8s.io build -t myapp:local .

# 然后在 Pod spec 中指定 imagePullPolicy: Never 即可使用本地镜像
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: myapp
    image: myapp:local
    imagePullPolicy: Never
EOF

六、可选：安装 BuildKit（支持 nerdctl build）

nerdctl 构建镜像需要 BuildKit daemon。

BUILDKIT_VERSION=$(curl -s https://api.github.com/repos/moby/buildkit/releases/latest \
  | grep tag_name | cut -d '"' -f4)

curl -LO "https://github.com/moby/buildkit/releases/download/${BUILDKIT_VERSION}/buildkit-${BUILDKIT_VERSION}.linux-amd64.tar.gz"

sudo tar Cxzvf /usr/local buildkit-${BUILDKIT_VERSION}.linux-amd64.tar.gz

# 创建 systemd 服务
sudo tee /etc/systemd/system/buildkit.service > /dev/null <<EOF
[Unit]
Description=BuildKit
After=network.target containerd.service

[Service]
ExecStart=/usr/local/bin/buildkitd \
  --addr unix:///run/buildkit/buildkitd.sock \
  --containerd-worker-addr /run/k3s/containerd/containerd.sock
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now buildkit

Refs

Mouser：轻量开源的罗技鼠标驱动替代方案

发表评论

以下内容整理自：https://meta.appinn.net/t/topic/83933

项目地址：https://github.com/TomBadash/Mouser

之前曾分享过一个可以按需精简安装 Logitech Options+ 功能的脚本工具 tjsky/logi-options-plus-mini，算是治标之策。然而实际使用下来，即便精简到了极致，Options+ 依然有个”顽疾”：它每周自动下载小几百 MB 的更新文件，却从不删除旧副本，硬盘俨然成了它的私人垃圾场。实测三个月后，相关文件体积已经膨胀到惊人的 2 GB。

于是，找到了一个真正的替代方案——Mouser。

简介

Mouser 是一个极其轻量、开源、完全本地运行的 Logitech Options+ 替代品，专用于对罗技 HID++ 鼠标进行按键和手势的重映射。目前对 MX Master 系列的支持最为完善，同时也对其他罗技型号提供了早期识别和通用 UI。

划重点：无需云端、无需罗技账号、纯本地运行！

安装包仅约 50 MB——而罗技官方完整离线驱动安装包高达 1.1 GB，半条命 2 整个游戏也才 1 GB……

功能特性

🖱️ 按键重映射

  • 可重映射中键、手势键、前进/后退、模式切换（滚轮按压）、水平滚轮等所有可编程按键
  • 按应用切换配置：在 Chrome 和 VS Code 之间切换时自动切换映射
  • 自定义快捷键：可将任意组合键（如 Ctrl+Shift+P）设为按键动作
  • 内置 30+ 常用动作，覆盖导航、浏览器、编辑、媒体、桌面等场景

⚙️ 设备控制

  • DPI / 指针速度：200–8000 DPI 滑块与快捷预设，实时同步
  • Smart Shift 开关：控制罗技”棘轮 ↔ 自由滚动”自动切换
  • 滚动方向反转：垂直/水平滚动可分别独立反转
  • 手势键 + 方向滑动：轻点触发一个动作，上/下/左/右滑动各自绑定不同动作

🖥️ 跨平台支持

  • 支持 Windows / macOS / Linux，各平台使用原生 Hook
  • 可开机自启，并支持”启动后最小化到托盘”
  • 系统托盘常驻，后台安静运行

🛡️ 隐私优先

  • 配置以本地 JSON 文件保存
  • 零遥测 / 零云端 / 零账号

当前支持的设备

鼠标型号 自动 HID++ 检测 UI 界面
MX Master 4 / 3S / 3 / 2S / MX Master 支持 MX Master 系列专用交互界面
MX Anywhere 3S / 3 / 2S 支持 通用界面（实验性专用界面，需手动切换）
MX Vertical 支持 通用界面（实验性专用界面，需手动切换）
其他罗技 HID++ 鼠标 部分支持（基于 PID/名称） 通用界面

下载与安装

无需安装，下载解压双击运行即可。

  1. 前往 最新 Release 页面 下载
  2. 根据系统选择：Mouser-Windows.zip / Mouser-macOS.zip / Mouser-Linux.zip
  3. 解压到任意目录
  4. ⚠️ 关键步骤：彻底关闭后台的 Logitech Options+！（两者同时抢占 HID++ 访问会一起崩溃）
  5. 运行：双击 Mouser.exe（Windows）/ Mouser.app（macOS）/ ./Mouser（Linux）

启动后，托盘区会出现图标，按键重映射立即生效。关闭窗口不会退出程序，如需完全退出请右键托盘图标选择 Quit Mouser。

macOS 用户首次运行需授予两个隐私权限（辅助功能 + 输入监控）；Linux 用户需要 /dev/input/event* 读取权限和 /dev/uinput 写入权限。

使用注意事项

  • Windows SmartScreen 拦截：点击”更多信息” → “仍要运行”即可
  • 配置文件位置：%APPDATA%\Mouser（Windows）、~/Library/Application Support/Mouser（macOS）、~/.config/Mouser（Linux）
  • 平时无需管理员权限；若游戏或系统级弹窗下侧键映射失效，可尝试以管理员权限运行
  • MX Anywhere / MX Vertical 系列可能需要在右上角手动切换交互式配置图，目前属实验性支持

平台替代方案参考

  • macOS 用户可以考虑付费但体验更好的 SteerMouse 或 BetterMouse，平滑滚动更接近苹果原生体验
  • Linux 用户推荐 LogiOps，无 GUI 但效果更好

小结

如果你是 MX Master 或 MX Anywhere 系列用户，只需要按键映射和 DPI 调节等基础功能，Mouser 是一个值得尝试的轻量替代方案，可以彻底告别臃肿的 Logitech Options+。

Claude Opus 4.7：优缺点与评测信息汇总

发表评论

以下内容转载自：https://linux.do/t/topic/1984117

基本资料

官方文：https://www.anthropic.com/news/claude-opus-4-7 官方文档：https://platform.claude.com/docs/en/about-claude/models/whats-new-claude-4-7 官方模型卡：https://cdn.sanity.io/files/4zrzovbb/website/037f06850df7fbe871e206dad004c3db5fd50340.pdf

价格：输入$5/输出$25与4.6一样 上下文：1m，与4.6一样 最大输出：128k，与4.6一样 输出速度：50tps左右，比4.6快30% 4.7 vs mythos：性能提升但明显不如mythos 切换回4.6的方法：/model claude-opus-4-6[1m]

优点：自主编程能力显著提升

能搞定更难、更长链的任务，还会主动验证输出再汇报。

优点：视觉处理能力显著提升

支持长边最高2576像素，即2.5k，是4.6三倍

新功能：/ultrareview

在线review，额外收费，Pro和Max用户可以免费试用三次

新功能：auto模式，自主决策、连续执行

首次向Max用户开放

新功能：在high和max之间增加xhigh

个人感觉实用性没多大，更多是一种营销策略，用来对用户认知作战，证明max比xhigh(映射codex)大 eb3ef932161795bdd77360f683519808|690x388

缺点：费token

因为分词器调整，4.7比4.6多费Token 来源：https://linux.do/t/topic/1983983 image|472x500

中文多费6%，英文多费59%，python多费21%，中国用户也别偷笑，你的代码还是英文的 image|332x500

官方说max会员额度可能有增加，对应分词器问题 来源：https://linux.do/t/topic/1984150 image|587x275

缺点：长上下文注意力差

MRCR长上下文测试，1m上下文性能比4.6有了大幅下降 来源：https://www.reddit.com/r/ClaudeAI/comments/1sn6eyd/ https://linux.do/t/topic/1983983/ https://linux.do/t/topic/1983616/

缺点：说话风格像GPT

https://linux.do/t/topic/1984103/

评测

官方自评

LMArena评分 截止2026-04-17 01:54尚未上线

来源：https://arena.ai/leaderboard

LiveBench评分

来源：https://livebench.ai 暂无opus-4.7-max

ArtificialAnalysis评分 截止2026-04-17 01:54尚未上线

来源：https://artificialanalysis.ai/

ARC-AGI评分

来源：https://arcprize.org/leaderboard

Humanity’s Last Exam (HLE)

门萨智商测试，未上线

来源：https://www.trackingai.org/home

vals.ai评分

Linux 裸机安全部署 Hermes Agent

发表评论

第一步：准备服务器环境

出于安全考虑，我们不应直接使用 root 用户运行应用程序。第一步是创建一个专用的非特权用户，并给予其 sudo 权限。

  1. 以 root 用户 SSH 登录你的服务器：
ssh root@your_server_ip
  1. 创建一个新用户（我们称之为 hermes）：
adduser hermes

系统会提示你为新用户设置密码和其他信息。

  1. 将新用户添加到 sudo 组，以便执行需要管理员权限的操作：
usermod -aG sudo hermes
  1. 切换到新用户：
su - hermes
  1. 更新系统软件包：
sudo apt update && sudo apt upgrade -y

现在，我们所有的操作都将在 hermes 用户下进行。

第二步：安装 hermes agent

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

第三步：配置 hermes agent

hermes setup

Refs

openFuyao NPU-Operator故障排查

发表评论

故障 pod describe

[root@master1 ~]# kubectl -n kube-system describe pod ascend-device-plugin-ll46f 
Name:                 ascend-device-plugin-ll46f
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      ascend-device-plugin-sa
Node:                 master1/10.17.30.131
Start Time:           Mon, 30 Mar 2026 11:08:32 +0800
Labels:               app.kubernetes.io/managed-by=npu-operator
                      controller-revision-hash=7df5dcb887
                      helm.sh/chart=npu-operator-0.15.0
                      name=ascend-device-plugin-ds
                      pod-template-generation=1
Annotations:          cni.projectcalico.org/containerID: c1f2adcaeaaf2bdcf0a6e09730f68231a293074e31d58f61997f714dfb520878
                      cni.projectcalico.org/podIP: 192.168.137.118/32
                      cni.projectcalico.org/podIPs: 192.168.137.118/32
                      scheduler.alpha.kubernetes.io/critical-pod: 
                      seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status:               Running
IP:                   192.168.137.118
IPs:
  IP:           192.168.137.118
Controlled By:  DaemonSet/ascend-device-plugin
Init Containers:
  init-permission:
    Container ID:  containerd://4406968a522bea48dfefebae81ec53644312762af4781c25de689952ed6c2d27
    Image:         cr.openfuyao.cn/openfuyao/busybox:1.36.1
    Image ID:      cr.openfuyao.cn/openfuyao/busybox@sha256:4b8407fadd8100c61b097d63efe992b2c033e7d371c9117f7a9462fe87e31176
    Port:          
<none>
    Host Port:     
<none>
    Command:
      sh
      -c
      chown 9000:9000 /var/log/mindx-dl /var/log/mindx-dl/devicePlugin
      chmod 750 /var/log/mindx-dl/devicePlugin

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 30 Mar 2026 15:28:32 +0800
      Finished:     Mon, 30 Mar 2026 15:28:32 +0800
    Ready:          True
    Restart Count:  1
    Environment:    
<none>
    Mounts:
      /var/log/mindx-dl/devicePlugin from log-path (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gfldg (ro)
Containers:
  device-plugin-01:
    Container ID:  containerd://fcc0c4742285847e2621a9a9217502307fc7e28644fbf86b32f9c11d67a2c0ab
    Image:         cr.openfuyao.cn/openfuyao/ascend-image/ascend-k8sdeviceplugin:v6.0.0
    Image ID:      cr.openfuyao.cn/openfuyao/ascend-image/ascend-k8sdeviceplugin@sha256:a5b9612b21bcd35384f9f19a05b2d7915b865e7b2be6a30bfd7806a9b8a86f58
    Port:          
<none>
    Host Port:     
<none>
    Command:
      /bin/bash
      -c
      --
    Args:
      device-plugin  -useAscendDocker=true -volcanoType=false -logFile=/var/log/mindx-dl/devicePlugin/devicePlugin.log -logLevel=0
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 31 Mar 2026 10:28:58 +0800
      Finished:     Tue, 31 Mar 2026 10:28:58 +0800
    Ready:          False
    Restart Count:  274
    Limits:
      cpu:     500m
      memory:  500Mi
    Requests:
      cpu:     500m
      memory:  500Mi
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /tmp from tmp (rw)
      /usr/local/Ascend/driver from hiai-driver (ro)
      /var/lib/kubelet/device-plugins from device-plugin (rw)
      /var/lib/kubelet/pod-resources from pod-resource (rw)
      /var/log/mindx-dl/devicePlugin from log-path (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gfldg (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  device-plugin:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/device-plugins
    HostPathType:  
  pod-resource:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pod-resources
    HostPathType:  
  hiai-driver:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/Ascend/driver
    HostPathType:  
  log-path:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/mindx-dl/devicePlugin
    HostPathType:  DirectoryOrCreate
  tmp:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp
    HostPathType:  
  kube-api-access-gfldg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              openfuyao.com/npu.present=
Tolerations:                 CriticalAddonsOnly op=Exists
                             device-plugin=v2:NoSchedule
                             huawei.com/Ascend910:NoSchedule op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Normal   Pulled   16m (x205 over 18h)     kubelet  (combined from similar events): Successfully pulled image "cr.openfuyao.cn/openfuyao/ascend-image/ascend-k8sdeviceplugin:v6.0.0" in 403ms (403ms including waiting). Image size: 48017174 bytes.
  Warning  BackOff  2m47s (x5216 over 18h)  kubelet  Back-off restarting failed container device-plugin-01 in pod ascend-device-plugin-ll46f_kube-system(8edcd384-ab2d-4998-8077-5ac58801c79e)
  Normal   Pulling  66s (x227 over 19h)     kubelet  Pulling image "cr.openfuyao.cn/openfuyao/ascend-image/ascend-k8sdeviceplugin:v6.0.0"

故障 pod /dev 检查

[root@master1 fuyao-26.3-rc3]# kubectl  -n kube-system exec -it daemonsets/ascend-device-plugin -- ls /dev
Defaulted container "device-plugin-01" out of: device-plugin-01, init-permission (init)
autofs           null               tty10  tty34  tty58    vcs5
bsg              ppp                tty11  tty35  tty59    vcs6
btrfs-control    ptmx               tty12  tty36  tty6     vcsa
bus              pts                tty13  tty37  tty60    vcsa1
core             random             tty14  tty38  tty61    vcsa2
cpu_dma_latency  raw                tty15  tty39  tty62    vcsa3
cuse             relationship_ctrl  tty16  tty4   tty63    vcsa4
davinci0         rfkill             tty17  tty40  tty7     vcsa5
davinci_manager  rtc0               tty18  tty41  tty8     vcsa6
devmm_svm        sda                tty19  tty42  tty9     vcsu
dri              sda1               tty2   tty43  ttyAMA0  vcsu1
fb0              sda2               tty20  tty44  ttyS0    vcsu2
fd               sg0                tty21  tty45  ttyS1    vcsu3
full             sg1                tty22  tty46  ttyS2    vcsu4
fuse             sg2                tty23  tty47  ttyS3    vcsu5
hidraw0          shm                tty24  tty48  uhid     vcsu6
hidraw1          snapshot           tty25  tty49  uinput   vfio
hisi_hdc         sr0                tty26  tty5   urandom  vga_arbiter
hwrng            sr1                tty27  tty50  usbmon0  vhost-net
input            stderr             tty28  tty51  usbmon1  vhost-vsock
kmsg             stdin              tty29  tty52  usbmon2  vport2p1
loop-control     stdout             tty3   tty53  vcs      zero
mapper           termination-log    tty30  tty54  vcs1
mem              tty                tty31  tty55  vcs2
mqueue           tty0               tty32  tty56  vcs3
net              tty1               tty33  tty57  vcs4

故障 pod 驱动检查

[root@master1 fuyao-26.3-rc3]# kubectl  -n kube-system exec -it daemonsets/ascend-device-plugin -- ls -lha /usr/local/Ascend/driver
Defaulted container "device-plugin-01" out of: device-plugin-01, init-permission (init)
total 44K
drwxr-xr-x  8 root root 4.0K Mar 27 08:03 .
drwxr-xr-x  3 root root 4.0K Mar 31 02:34 ..
drwxr-xr-x  2 root root 4.0K Mar 27 08:01 bin
-r--r--r--  1 root root   20 Mar 27 08:01 build.info
dr-xr-x---  2 root root 4.0K Mar 27 08:01 device
dr-x------ 41 root root 4.0K Mar 27 08:01 kernel
drwxr-xr-x  6 root root 4.0K Mar 27 08:01 lib64
-r--r-----  1 root root   56 Mar 27 08:01 scene.info
dr-xr-x---  2 root root 4.0K Mar 27 08:01 script
drwxr-xr-x  2 root root 4.0K Mar 27 08:01 tools
-r--r--r--  1 root root  352 Mar 27 08:03 version.info

故障 pod 日志

[root@master1 ~]# kubectl -n kube-system logs daemonsets/ascend-device-plugin --previous
Defaulted container "device-plugin-01" out of: device-plugin-01, init-permission (init)
[INFO]     2026/03/31 06:46:54.593254 1       hwlog/api.go:108    devicePlugin.log's logger init success
[INFO]     2026/03/31 06:46:54.593449 1       main.go:187    ascend device plugin starting and the version is v6.0.0_linux-aarch64
[INFO]     2026/03/31 06:46:54.593494 1       main.go:188    ascend device plugin starting scene is center
[INFO]     2026/03/31 06:46:54.787930 1       devmanager/devmanager.go:104    the dcmi version is 24.1.rc3
[ERROR]    2026/03/31 06:46:54.788019 1       devmanager/devmanager.go:211    get error card quantity: 0
[ERROR]    2026/03/31 06:46:54.788052 1       devmanager/devmanager.go:195    get card list failed for init
[ERROR]    2026/03/31 06:46:54.788101 1       main.go:203    init devmanager failed, err: auto init failed, err: get card list failed for init

故障 pod 驱动检查

[root@master1 ~]# kubectl -n kube-system exec -it daemonsets/ascend-device-plugin -- bash -c 'find /usr/local/Ascend/driver -name libdcmi.so 2>/dev/null; echo $LD_LIBRARY_PATH'
Defaulted container "device-plugin-01" out of: device-plugin-01, init-permission (init)
/usr/local/Ascend/driver/lib64/driver/libdcmi.so
command terminated with exit code 137
[root@master1 ~]# ps -ef | grep -E 'dmp_daemon|slogd' | grep -v grep
root       21578       1  0 Mar30 ?        00:00:19 /usr/sbin/rsyslogd -n -i/var/run/rsyslogd.pid

检查服务状态？

[root@master1 ~]# systemctl status ascend-dmi
Unit ascend-dmi.service could not be found.
[root@master1 ~]# systemctl status ascend-dkms
Unit ascend-dkms.service could not be found.
[root@master1 ~]# systemctl status npu-smi
Unit npu-smi.service could not be found.
[root@master1 ~]# find / -name dmp_daemon 2>/dev/null
[root@master1 ~]# find / -name slogd 2>/dev/null
[root@master1 ~]# ls -l /var/dmp_daemon /var/slogd 2>/dev/null
[root@master1 ~]#

dcmi 问题，需硬件排查

检查卡获取

#include <stdlib.h>
#include <stdio.h>
#include "dcmi_interface_api.h"

int my_get_card_list();

int main(int argc,char *argv[])
{
    my_get_card_list();
    return 0;
}

int my_get_card_list()
{
    printf("\n==================================card id info list=========================\n");
    dcmi_init();
    int card_num = 0;
    int card_list[16] = {0};
    int ret = dcmi_get_card_list(&card_num, card_list, 16);
    if (ret != DCMI_OK) {
        printf("dcmi get card list failed ret=%d\n", ret);
    }
    printf("card_num=%d, card_list:[",card_num);
    for (int i = 0; i < card_num; i++) {
        printf("%d ", card_list[i]);
    }
}
cc ./test1.c -o test1 -I /usr/local/dcmi -L /usr/local/dcmi -ldcmi

-I头文件（.h）搜索路径
-L库文件（.so/.a）搜索路径
-l链接的库名（去掉 lib 前缀）
nerdctl run --rm \
  -v /usr/local/Ascend:/usr/local/Ascend \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v $(pwd):/build \
  ubuntu:18.04 bash -c "
    sed -i -e 's@http*://ports.ubuntu.com/\? @http://10.17.31.217:8081/repository/mirror-ubuntu-ports/@g' \
           -e 's@http*://ports.ubuntu.com@http://10.17.31.217:8081/repository/mirror-ubuntu-ports@g' \
           /etc/apt/sources.list
    apt update && apt install -y gcc
    cd /build
    cc ./test1.c -o test1 \
      -I /usr/local/dcmi \
      -L /usr/local/dcmi \
      -L /usr/local/Ascend/driver/lib64/common \
      -L /usr/local/Ascend/driver/lib64/driver \
      -ldcmi \
      -Wl,-rpath,/usr/local/Ascend/driver/lib64/common \
      -Wl,-rpath,/usr/local/Ascend/driver/lib64/driver \
      -Wl,-rpath,/usr/local/dcmi
  "

分析二进制：

[root@master1 ascend_debug]# ldd ./test1 | grep -i dcmi
        libdcmi.so => /usr/local/Ascend/driver/lib64/driver/libdcmi.so (0x0000ffffa6dd0000)
[root@master1 ascend_debug]# LD_DEBUG=libs ./test1 2>&1 | grep -i dcmi
    284830:     find library=libdcmi.so [0]; searching
    284830:      search path=/usr/local/Ascend/driver/lib64/common/tls/aarch64/atomics:/usr/local/Ascend/driver/lib64/common/tls/aarch64:/usr/local/Ascend/driver/lib64/common/tls/atomics:/usr/local/Ascend/driver/lib64/common/tls:/usr/local/Ascend/driver/lib64/common/aarch64/atomics:/usr/local/Ascend/driver/lib64/common/aarch64:/usr/local/Ascend/driver/lib64/common/atomics:/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64/driver/tls/aarch64/atomics:/usr/local/Ascend/driver/lib64/driver/tls/aarch64:/usr/local/Ascend/driver/lib64/driver/tls/atomics:/usr/local/Ascend/driver/lib64/driver/tls:/usr/local/Ascend/driver/lib64/driver/aarch64/atomics:/usr/local/Ascend/driver/lib64/driver/aarch64:/usr/local/Ascend/driver/lib64/driver/atomics:/usr/local/Ascend/driver/lib64/driver:/usr/local/dcmi/tls/aarch64/atomics:/usr/local/dcmi/tls/aarch64:/usr/local/dcmi/tls/atomics:/usr/local/dcmi/tls:/usr/local/dcmi/aarch64/atomics:/usr/local/dcmi/aarch64:/usr/local/dcmi/atomics:/usr/local/dcmi            (RUNPATH from file ./test1)
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/tls/aarch64/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/tls/aarch64/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/tls/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/tls/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/aarch64/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/aarch64/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/common/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/tls/aarch64/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/tls/aarch64/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/tls/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/tls/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/aarch64/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/aarch64/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/atomics/libdcmi.so
    284830:       trying file=/usr/local/Ascend/driver/lib64/driver/libdcmi.so
    284830:      search path=/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64/driver:/usr/local/dcmi/tls/aarch64/atomics:/usr/local/dcmi/tls/aarch64:/usr/local/dcmi/tls/atomics:/usr/local/dcmi/tls:/usr/local/dcmi/aarch64/atomics:/usr/local/dcmi/aarch64:/usr/local/dcmi/atomics:/usr/local/dcmi          (RUNPATH from file ./test1)
    284830:       trying file=/usr/local/dcmi/tls/aarch64/atomics/libc.so.6
    284830:       trying file=/usr/local/dcmi/tls/aarch64/libc.so.6
    284830:       trying file=/usr/local/dcmi/tls/atomics/libc.so.6
    284830:       trying file=/usr/local/dcmi/tls/libc.so.6
    284830:       trying file=/usr/local/dcmi/aarch64/atomics/libc.so.6
    284830:       trying file=/usr/local/dcmi/aarch64/libc.so.6
    284830:       trying file=/usr/local/dcmi/atomics/libc.so.6
    284830:       trying file=/usr/local/dcmi/libc.so.6
    284830:     calling init: /usr/local/Ascend/driver/lib64/driver/libdcmi.so
    284830:     calling fini: /usr/local/Ascend/driver/lib64/driver/libdcmi.so [0]
[root@master1 ascend_debug]# readlink -f /usr/local/dcmi/libdcmi.so
/usr/local/dcmi/libdcmi.so
[root@master1 ascend_debug]# readlink -f /usr/local/Ascend/driver/lib64/driver/libdcmi.so
/usr/local/Ascend/driver/lib64/driver/libdcmi.so
[root@master1 ascend_debug]# sha256sum /usr/local/dcmi/libdcmi.so /usr/local/Ascend/driver/lib64/driver/libdcmi.so
13a38cae84bad0f06367ff9280016e372c0608ca16465b5ae5f000d3844ee401  /usr/local/dcmi/libdcmi.so
13a38cae84bad0f06367ff9280016e372c0608ca16465b5ae5f000d3844ee401  /usr/local/Ascend/driver/lib64/driver/libdcmi.so

跟踪 strace

宿主机跑
strace -f -o /tmp/host.strace -e trace=file,ioctl ./test1
容器里跑
strace -f -o /tmp/container.strace -e trace=file,ioctl ./test1

新增挂载继续跟踪

volumeMounts:
        - name: hdc-basic
          mountPath: /etc/hdcBasic.cfg
          readOnly: true
        - name: localtime
          mountPath: /etc/localtime
          readOnly: true
      volumes:
      - name: hdc-basic
        hostPath:
          path: /etc/hdcBasic.cfg
          type: File
      - name: localtime
        hostPath:
          path: /etc/localtime
          type: File

跟踪并查看日志

kubectl -n kube-system exec -it ascend-device-plugin-69q5t -c device-plugin-01 -- bash

strace -f -o /tmp/container.strace -e trace=file,ioctl ./test1

root@ascend-device-plugin-69q5t:/tmp# strace -f -o /tmp/container.strace -e trace=file,ioctl ./test1

==================================card id info list=========================
card_num=0, card_list:[

root@ascend-device-plugin-69q5t:/tmp# cat /var/log/nputools_LOG_INFO.log > /tmp/nputools_LOG_INFO.log
root@ascend-device-plugin-69q5t:/tmp# cat /var/log/nputools_LOG_ERR.log > /tmp/nputools_LOG_ERR.log
cat: /var/log/nputools_LOG_ERR.log: No such file or directory
root@ascend-device-plugin-69q5t:/tmp# cat /tmp/nputools_LOG_INFO.log 
[2026/04/01 11:18:12][0583][root][127.0.0.1][dcmi_api.c,dcmi_board_init,86]:dcmi board init success. device_count=1.
[2026/04/01 11:18:12][0583][root][127.0.0.1][dcmi_api.c,dcmi_init,119]:dcmi init all success.

检查代码2

#include <stdio.h>
#include <stdlib.h>
#include "dcmi_interface_api.h"

#ifndef DCMI_OK
#define DCMI_OK 0
#endif

/* 头文件里没看到这个声明，手动补一个 */
extern int dcmi_get_card_num_list(int *card_num, int *card_list, int list_length);

static void print_list(const char *name, int ret, int num, int *list) {
    printf("%s ret=%d num=%d list=[", name, ret, num);
    for (int i = 0; i < num; ++i) {
        printf("%d ", list[i]);
    }
    printf("]\n");
}

int main(void) {
    int ret = dcmi_init();
    printf("dcmi_init ret=%d\n", ret);
    if (ret != DCMI_OK) {
        return 1;
    }

    int card_num = 0;
    int card_list[16] = {0};

    ret = dcmi_get_card_list(&card_num, card_list, 16);
    print_list("dcmi_get_card_list", ret, card_num, card_list);

    int card_num2 = 0;
    int card_list2[16] = {0};
    ret = dcmi_get_card_num_list(&card_num2, card_list2, 16);
    print_list("dcmi_get_card_num_list", ret, card_num2, card_list2);

    for (int i = 0; i < card_num && i < 16; ++i) {
        int dev_num = -1;
        ret = dcmi_get_device_num_in_card(card_list[i], &dev_num);
        printf("dcmi_get_device_num_in_card card=%d ret=%d dev_num=%d\n",
               card_list[i], ret, dev_num);
    }

    return 0;
}

主机编译

cc ./test2.c -o test2 -I /usr/local/dcmi -L /usr/local/dcmi -ldcmi

容器编译

nerdctl run --rm \
  -v /usr/local/Ascend:/usr/local/Ascend \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v $(pwd):/build \
  ubuntu:18.04 bash -c "
    sed -i -e 's@http*://ports.ubuntu.com/\? @http://10.17.31.217:8081/repository/mirror-ubuntu-ports/@g' \
           -e 's@http*://ports.ubuntu.com@http://10.17.31.217:8081/repository/mirror-ubuntu-ports@g' \
           /etc/apt/sources.list
    apt update && apt install -y gcc
    cd /build
    cc ./test2.c -o test2 \
      -I /usr/local/dcmi \
      -L /usr/local/dcmi \
      -L /usr/local/Ascend/driver/lib64/common \
      -L /usr/local/Ascend/driver/lib64/driver \
      -ldcmi \
      -Wl,-rpath,/usr/local/Ascend/driver/lib64/common \
      -Wl,-rpath,/usr/local/Ascend/driver/lib64/driver \
      -Wl,-rpath,/usr/local/dcmi
  "

拷入容器运行

kubectl -n kube-system cp ./test2 ascend-device-plugin-69q5t:/tmp/
# 主机运行
[root@master1 ascend_debug]# ./test2
dcmi_init ret=0
dcmi_get_card_list ret=0 num=1 list=[176 ]
dcmi_get_card_num_list ret=0 num=1 list=[176 ]
dcmi_get_device_num_in_card card=176 ret=0 dev_num=1

# 容器运行
root@ascend-device-plugin-69q5t:/tmp# ./test2 
dcmi_init ret=0
dcmi_get_card_list ret=0 num=0 list=[]
dcmi_get_card_num_list ret=0 num=0 list=[]

虚拟机场景

经过许老师认真定位，最终发现是因为非裸金属环境。虚拟机场景需要定制镜像。

根据官网文档：

如果在虚拟机场景下部署Ascend Device Plugin，需要在Ascend Device Plugin的镜像中安装systemd，推荐在Dockerfile中加入RUN apt-get update && apt-get install -y systemd命令进行安装。

为了使用 nerdctl 构建镜像首先安装 buildkit

wegt https://github.com/moby/buildkit/releases/download/v0.29.0/buildkit-v0.29.0.linux-arm64.tar.gz
tar zxvf buildkit-v0.29.0.linux-arm64.tar.gz
cp bin/* /usr/local/bin/

之后找一个新终端启动 buildkit ，这里是为了 nerdctl 构建 image, 如果不需要则不用启动。

buildkitd --oci-worker=false --containerd-worker=true --containerd-worker-namespace=k8s.io

Dockerfile 如下：

镜像源部分按需修改


FROM hub.oepkgs.net/openfuyao/ascendhub/ascend-k8sdeviceplugin:v6.0.0

替换 apt 镜像源

RUN sed -i \ -e ‘s@http://ports.ubuntu.com/\? @http://10.17.31.217:8081/repository/mirror-ubuntu-ports/@g‘ \ -e ‘s@http://ports.ubuntu.com@http://10.17.31.217:8081/repository/mirror-ubuntu-ports@g‘ \ /etc/apt/sources.list

安装 systemd

RUN apt-get update && \ apt-get install -y –no-install-recommends systemd systemd-sysv && \ apt-get clean && \ rm -rf /var/lib/apt/lists/*

设置 systemd 为 init

STOPSIGNAL SIGRTMIN+3


> STOPSIGNAL SIGRTMIN+3 是告诉容器运行时（containerd/docker）停止这个容器时应该发送哪个信号。
> 如果你的容器里 不跑 systemd 作为主进程（比如 entrypoint 是业务程序），这行可以删掉，没有任何作用。
> 如果确实用 systemd 管理容器内服务，保留它能避免 kubectl delete pod 时等待 30 秒超时再强杀的问题。

构建命令如下：

nerdctl build \ –namespace k8s.io \ -t hub.oepkgs.net/openfuyao/ascendhub/ascend-k8sdeviceplugin:v6.0.0-systemd \ -f Dockerfile \ .


之后将出问题的镜像替换为新构建的镜像即可。

npu-operator 有同样的问题，一样修改即可。

## 修复确认
> 最终在 node 中能看到 npu 资源即成功。

[root@master1 ~]# kubectl describe node master1 Name: master1 Roles: control-plane,master,node,worker Labels: accelerator=huawei-Ascend310P beta.kubernetes.io/arch=arm64 beta.kubernetes.io/os=linux … servertype=Ascend310P-8 workerselector=dls-worker-node Annotations: baseDeviceInfos: {“Ascend310P-0”:{“IP”:””,”SuperDeviceID”:0}} … Capacity: cpu: 16 ephemeral-storage: 129724184Ki huawei.com/Ascend310P: 1 hugepages-1Gi: 0 hugepages-2Mi: 0 hugepages-32Mi: 0 hugepages-64Ki: 0 memory: 32595632Ki pods: 110 Allocatable: cpu: 16 ephemeral-storage: 119553807777 huawei.com/Ascend310P: 1 hugepages-1Gi: 0 hugepages-2Mi: 0 hugepages-32Mi: 0 hugepages-64Ki: 0 memory: 32493232Ki pods: 110 … Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits

cpu 15565m (97%) 16910m (105%) memory 17492Mi (55%) 30900Mi (97%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) hugepages-32Mi 0 (0%) 0 (0%) hugepages-64Ki 0 (0%) 0 (0%) huawei.com/Ascend310P 0 0 …

openFuyao 2603 共测测试报告

发表评论

相关链接

测试环境

  • CPU: Kunpeng-920
  • OS: openEuler 24.03 LTS SP3 aarch64
  • Fuyao Version: v26.03 rc3
  • docker: 2:18.09.0-346.oe2403sp3

测试特性

  • 在线部署；
  • 离线包制备；
  • 离线部署；
  • 安装部署前置检查工具；
  • NPU Operator；
  • AI推理套件；

    建议优化点

  • 环境检测工具，检查 iptables 默认策略是否放行，若未放行可能在部署成功后无法访问；默认防火墙策略为 FORWARD DROP ，对集群运行和访问带来的潜在问题；
  • 运行 cli 前检查是否存在命令并及时抛出错误；检查 tar / unzip 是否安装，安装过程有很多地方会用到，而且出错时不会得到明显的解压失败报错，难以定位问题。
  • 安装命令变化，考虑上下兼容性？

    场景记录

离线部署管理面和业务面集群

  • CPU: Kunpeng-920
  • OS: openEuler 24.03 LTS SP3 aarch64
  • Fuyao Version: v26.03 rc3

  • docker: 2:18.09.0-346.oe2403sp3

    arm64 环境下构建离线制品包为什么会执行 amd64 的 bin

    [bke][2026-03-26 07:48:49][INFO] The bke binary file version is . sh: line 1: /root/fuyao-26-03/packages/usr/bin/bkeadm_linux_amd64: cannot execute binary file: Exec format error

完整日志

 [root@master1 fuyao-26-03]# cat build-offline-package.log | grep -v sha256 
2026-03-26T06:49:56.467+0800    info    infrastructure/infrastructure.go:53     The docker client is ready.
[bke][2026-03-26 06:49:56][step.1] Configuration file check
[bke][2026-03-26 06:49:56][step.2] Creates a workspace in the current directory
[bke][2026-03-26 06:49:56][step.5] Collect the required image files
[bke][2026-03-26 06:49:56][INFO] Try pulling away the mirror image cr.openfuyao.cn/openfuyao/registry:2.8.1
[bke][2026-03-26 06:49:56][step.3] Collect host dependency packages and package files
[bke][2026-03-26 06:49:56][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/kubernetes/kubernetes/releases/download/1.34.3-of.1/bin/linux/arm64/kubectl to /root/fuyao-26-03/packages/tmp/packages/files/kubectl-v1.34.3-of.1-arm64
[bke][2026-03-26 06:50:07][INFO] Try pulling away the mirror image cr.openfuyao.cn/openfuyao/registry:2.8.1
[bke][2026-03-26 06:50:18][step.6] Collect images from the source repository to the target repository
[bke][2026-03-26 06:50:18][INFO] Remove the image repository
2026-03-26T06:50:18.026+0800    info    infrastructure/infrastructure.go:53     The docker client is ready.
2026-03-26T06:50:18.028+0800    info    infrastructure/infrastructure.go:53     The docker client is ready.
[bke][2026-03-26 06:50:20][WARN] Image cr.openfuyao.cn/openfuyao/registry:2.8.1 inspect failed: Error response from daemon: no such image: cr.openfuyao.cn/openfuyao/registry:2.8.1: No such image: cr.openfuyao.cn/openfuyao/registry:2.8.1, retrying (1/3)...
[bke][2026-03-26 06:50:21][WARN] Image cr.openfuyao.cn/openfuyao/registry:2.8.1 inspect failed: Error response from daemon: no such image: cr.openfuyao.cn/openfuyao/registry:2.8.1: No such image: cr.openfuyao.cn/openfuyao/registry:2.8.1, retrying (2/3)...
[bke][2026-03-26 06:50:22][WARN] Image cr.openfuyao.cn/openfuyao/registry:2.8.1 inspect failed: Error response from daemon: no such image: cr.openfuyao.cn/openfuyao/registry:2.8.1: No such image: cr.openfuyao.cn/openfuyao/registry:2.8.1, retrying (3/3)...
[bke][2026-03-26 06:50:23][WARN] Get image cr.openfuyao.cn/openfuyao/registry:2.8.1 inspect failed: failed to inspect image cr.openfuyao.cn/openfuyao/registry:2.8.1 after 3 attempts: Error response from daemon: no such image: cr.openfuyao.cn/openfuyao/registry:2.8.1: No such image: cr.openfuyao.cn/openfuyao/registry:2.8.1
[bke][2026-03-26 06:50:23][INFO] Image cr.openfuyao.cn/openfuyao/registry:2.8.1 is downloading
[bke][2026-03-26 06:50:37][INFO] Wait for the container mirroring service to start...
[bke][2026-03-26 06:50:42][INFO] The container mirroring service is started. 
Getting image list signatures
Copying 6 images generated from 6 images in list
Getting image source signatures
[bke][2026-03-26 06:50:56][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/kubernetes/kubernetes/releases/download/1.34.3-of.1/bin/linux/arm64/kubelet to /root/fuyao-26-03/packages/tmp/packages/files/kubelet-v1.34.3-of.1-arm64
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 7 images generated from 7 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
[bke][2026-03-26 06:52:10][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/kubernetes/kubernetes/releases/download/1.34.3-of.1/bin/linux/amd64/kubectl to /root/fuyao-26-03/packages/tmp/packages/files/kubectl-v1.34.3-of.1-amd64
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
[bke][2026-03-26 06:53:23][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/kubernetes/kubernetes/releases/download/1.34.3-of.1/bin/linux/amd64/kubelet to /root/fuyao-26-03/packages/tmp/packages/files/kubelet-v1.34.3-of.1-amd64
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
[bke][2026-03-26 06:54:34][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/containerd/containerd/releases/download/v2.1.1-origin/containerd-v2.1.1-linux-amd64.tar.gz to /root/fuyao-26-03/packages/tmp/packages/files/containerd-v2.1.1-linux-amd64.tar.gz
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 3 images generated from 3 images in list
Getting image source signatures
[bke][2026-03-26 06:55:53][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/containerd/containerd/releases/download/v2.1.1-origin/containerd-v2.1.1-linux-arm64.tar.gz to /root/fuyao-26-03/packages/tmp/packages/files/containerd-v2.1.1-linux-arm64.tar.gz
Writing manifest to image destination
Getting image source signatures
[bke][2026-03-26 06:57:07][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/containernetworking/plugins/releases/download/v1.4.1/cni-plugins-linux-amd64-v1.4.1.tgz to /root/fuyao-26-03/packages/tmp/packages/files/cni-plugins-linux-amd64-v1.4.1.tgz
[bke][2026-03-26 06:58:00][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/containernetworking/plugins/releases/download/v1.4.1/cni-plugins-linux-arm64-v1.4.1.tgz to /root/fuyao-26-03/packages/tmp/packages/files/cni-plugins-linux-arm64-v1.4.1.tgz
Writing manifest to image destination
Getting image source signatures
[bke][2026-03-26 06:58:49][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/helm/releases/download/v3.14.2/helm-v3.14.2-linux-amd64.tar.gz to /root/fuyao-26-03/packages/tmp/packages/files/helm-v3.14.2-linux-amd64.tar.gz
[bke][2026-03-26 06:59:08][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/helm/releases/download/v3.14.2/helm-v3.14.2-linux-arm64.tar.gz to /root/fuyao-26-03/packages/tmp/packages/files/helm-v3.14.2-linux-arm64.tar.gz
[bke][2026-03-26 06:59:24][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/mikefarah/yq/releases/download/v4.43.1/yq_linux_arm64 to /root/fuyao-26-03/packages/tmp/packages/files/yq_linux_arm64
[bke][2026-03-26 06:59:35][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/mikefarah/yq/releases/download/v4.43.1/yq_linux_amd64 to /root/fuyao-26-03/packages/tmp/packages/files/yq_linux_amd64
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 10 images generated from 10 images in list
Getting image source signatures
[bke][2026-03-26 06:59:44][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/jqlang/jq/releases/download/v1.7.1/jq-linux-arm64 to /root/fuyao-26-03/packages/tmp/packages/files/jq-linux-arm64
[bke][2026-03-26 06:59:46][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/jqlang/jq/releases/download/v1.7.1/jq-linux-amd64 to /root/fuyao-26-03/packages/tmp/packages/files/jq-linux-amd64
[bke][2026-03-26 06:59:49][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl-certinfo_1.6.4_linux_arm64 to /root/fuyao-26-03/packages/tmp/packages/files/cfssl-certinfo_1.6.4_linux_arm64
[bke][2026-03-26 07:00:00][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl-certinfo_1.6.4_linux_amd64 to /root/fuyao-26-03/packages/tmp/packages/files/cfssl-certinfo_1.6.4_linux_amd64
[bke][2026-03-26 07:00:11][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_arm64 to /root/fuyao-26-03/packages/tmp/packages/files/cfssl_1.6.4_linux_arm64
[bke][2026-03-26 07:00:24][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64 to /root/fuyao-26-03/packages/tmp/packages/files/cfssl_1.6.4_linux_amd64
[bke][2026-03-26 07:00:38][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_arm64 to /root/fuyao-26-03/packages/tmp/packages/files/cfssljson_1.6.4_linux_arm64
[bke][2026-03-26 07:00:46][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64 to /root/fuyao-26-03/packages/tmp/packages/files/cfssljson_1.6.4_linux_amd64
[bke][2026-03-26 07:00:55][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/opencontainers/runc/releases/download/v1.1.12/runc-arm64 to /root/fuyao-26-03/packages/tmp/packages/files/runc-arm64
[bke][2026-03-26 07:01:07][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/opencontainers/runc/releases/download/v1.1.12/runc-amd64 to /root/fuyao-26-03/packages/tmp/packages/files/runc-amd64
[bke][2026-03-26 07:01:19][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/etcd-io/etcd/releases/download/v3.5.6/etcdctl-v3.5.6-linux-amd64 to /root/fuyao-26-03/packages/tmp/packages/files/etcdctl-v3.5.6-linux-amd64
[bke][2026-03-26 07:01:40][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/etcd-io/etcd/releases/download/v3.5.6/etcdctl-v3.5.6-linux-arm64 to /root/fuyao-26-03/packages/tmp/packages/files/etcdctl-v3.5.6-linux-arm64
Writing manifest to image destination
Getting image source signatures
[bke][2026-03-26 07:01:58][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/rpm/releases/download/v0.0.1/rpm.tar.gz to /root/fuyao-26-03/packages/tmp/packages/files/rpm.tar.gz
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 7 images generated from 7 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 8 images generated from 8 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
[bke][2026-03-26 07:45:30][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/nfs/nfsshare.tar.gz to /root/fuyao-26-03/packages/tmp/packages/files/nfsshare.tar.gz
[bke][2026-03-26 07:45:30][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/bkeadm/releases/download/1.2.1/bkeadm_linux_amd64 to /root/fuyao-26-03/packages/tmp/packages/files/bkeadm_linux_amd64
[bke][2026-03-26 07:46:48][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/bkeadm/releases/download/1.2.1/bkeadm_linux_arm64 to /root/fuyao-26-03/packages/tmp/packages/files/bkeadm_linux_arm64
[bke][2026-03-26 07:48:03][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/oauth-webhook-1.0.2.tgz to /root/fuyao-26-03/packages/tmp/charts/oauth-webhook-1.0.2.tgz
Writing manifest to image destination
[bke][2026-03-26 07:48:03][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/oauth-server-1.0.3.tgz to /root/fuyao-26-03/packages/tmp/charts/oauth-server-1.0.3.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/console-website-1.0.4.tgz to /root/fuyao-26-03/packages/tmp/charts/console-website-1.0.4.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/monitoring-service-1.0.4.tgz to /root/fuyao-26-03/packages/tmp/charts/monitoring-service-1.0.4.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/console-service-1.0.4.tgz to /root/fuyao-26-03/packages/tmp/charts/console-service-1.0.4.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/marketplace-service-1.0.3.tgz to /root/fuyao-26-03/packages/tmp/charts/marketplace-service-1.0.3.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/application-management-service-1.0.2.tgz to /root/fuyao-26-03/packages/tmp/charts/application-management-service-1.0.2.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/plugin-management-service-1.0.2.tgz to /root/fuyao-26-03/packages/tmp/charts/plugin-management-service-1.0.2.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/user-management-operator-1.0.2.tgz to /root/fuyao-26-03/packages/tmp/charts/user-management-operator-1.0.2.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/web-terminal-service-1.0.3.tgz to /root/fuyao-26-03/packages/tmp/charts/web-terminal-service-1.0.3.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/installer-service-1.2.1.tgz to /root/fuyao-26-03/packages/tmp/charts/installer-service-1.2.1.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/installer-website-1.2.1.tgz to /root/fuyao-26-03/packages/tmp/charts/installer-website-1.2.1.tgz
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/harbor-1.11.4.tgz to /root/fuyao-26-03/packages/tmp/charts/harbor-1.11.4.tgz
Getting image source signatures
[bke][2026-03-26 07:48:04][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/bke-console-website-1.0.3.tgz to /root/fuyao-26-03/packages/tmp/charts/bke-console-website-1.0.3.tgz
[bke][2026-03-26 07:48:05][INFO] Collecting file packages https://openfuyao.obs.cn-north-4.myhuaweicloud.com/charts/releases/download/bke-console-service-1.0.2.tgz to /root/fuyao-26-03/packages/tmp/charts/bke-console-service-1.0.2.tgz
[bke][2026-03-26 07:48:49][step.4] Collect the bke binary file
[bke][2026-03-26 07:48:49][INFO] The bke binary file version is . sh: line 1: /root/fuyao-26-03/packages/usr/bin/bkeadm_linux_amd64: cannot execute binary file: Exec format error
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 17 images generated from 17 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 16 images generated from 16 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 2 images generated from 2 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 10 images generated from 10 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 4 images generated from 4 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
Getting image list signatures
Copying 5 images generated from 5 images in list
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Getting image source signatures
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
[bke][2026-03-26 08:41:56][INFO] The system starts to pack the image file.
[bke][2026-03-26 08:43:49][INFO] Remove the image repository
2026-03-26T08:43:49.940+0800    info    infrastructure/infrastructure.go:53     The docker client is ready.
[bke][2026-03-26 08:43:51][step.7] Build the bke package, please wait for the larger package...
[bke][2026-03-26 08:46:22][step.8] Packaging complete bke.tar.gz

openEuler 24.03 lts 部署 npu operator 有 pod 无法启动：

Every 1.0s: kubectl  get pod -A                                                                                                                                                                                                                                                                                                                          master1: Fri Mar 27 10:33:04 2026

NAMESPACE                     NAME                                                 READY   STATUS             RESTARTS      AGE
cluster-system                bkeagent-deployer-fcnfd                              1/1     Running            0             29m
default                       mindio-acp-g67n9                                     1/1     Running            0             7m56s
default                       mindio-tft-bk8mj                                     1/1     Running            0             7m57s
ingress-nginx                 ingress-nginx-admission-create-fh4rv                 0/1     Completed          0             28m
ingress-nginx                 ingress-nginx-admission-patch-nn79l                  0/1     Completed          2             28m
ingress-nginx                 ingress-nginx-controller-jdb5s                       1/1     Running            0             28m
kube-system                   ascend-device-plugin-5hmhf                           0/1     CrashLoopBackOff   6 (33s ago)   7m59s
kube-system                   calico-kube-controllers-9c987b475-db86h              1/1     Running            0             29m
kube-system                   calico-node-sfvjm                                    1/1     Running            0             29m
kube-system                   coredns-6bd9b76578-2mqzh                             1/1     Running            2 (29m ago)   29m
kube-system                   coredns-6bd9b76578-94pmp                             1/1     Running            2 (21m ago)   29m
kube-system                   etcd-master1                                         1/1     Running            0             28m
kube-system                   kube-apiserver-master1                               1/1     Running            0             27m
kube-system                   kube-controller-manager-master1                      1/1     Running            1 (27m ago)   29m
kube-system                   kube-proxy-rwpzc                                     1/1     Running            0             29m
kube-system                   kube-scheduler-master1                               1/1     Running            1 (27m ago)   29m
kube-system                   metrics-server-db68b78d-b8kkn                        1/1     Running            0             26m
mindx-dl                      ascend-operator-manager-5d4d89f675-8dzn4             0/1     Pending            0             7m59s
mindx-dl                      clusterd-5588c5dc88-qmblm                            0/1     Pending            0             7m57s
mindx-dl                      noded-rc5lj                                          1/1     Running            0             7m59s
mindx-dl                      resilience-controller-8686bbd76f-qzc2z               0/1     Pending            0             7m57s
monitoring                    alertmanager-main-0                                  2/2     Running            0             28m
monitoring                    alertmanager-main-1                                  2/2     Running            0             28m
monitoring                    alertmanager-main-2                                  2/2     Running            0             28m
monitoring                    blackbox-exporter-6877f5c5f7-66qn4                   3/3     Running            0             28m
monitoring                    kube-state-metrics-6f6c47f5f4-4pt9s                  3/3     Running            0             28m
monitoring                    node-exporter-ng468                                  2/2     Running            0             28m
monitoring                    prometheus-k8s-0                                     2/2     Running            0             28m
monitoring                    prometheus-k8s-1                                     2/2     Running            0             28m
monitoring                    prometheus-operator-6698d7bc85-72xwx                 2/2     Running            0             28m
npu-exporter                  npu-exporter-vnzk6                                   0/1     Pending            0             7m57s
npu                           ascend-runtime-containerd-7s6jv                      1/1     Running            0             8m   
npu                           npu-driver-l7rjp                                     1/1     Running            0             8m   
npu                           npu-feature-discovery-xr7nt                          1/1     Running            0             19m
npu                           npu-node-feature-discovery-gc-5d97746dbc-wvnx6       1/1     Running            0             9m52s
npu                           npu-node-feature-discovery-master-664666b7bb-7qjfh   1/1     Running            0             9m43s
npu                           npu-node-feature-discovery-worker-8mj5w              1/1     Running            0             9m38s
npu                           npu-operator-5678cd59d4-ljvc8                        1/1     Running            0             19m
openfuyao-system-controller   modify-manifests-master1-rhfqf                       0/1     Completed          0             27m
openfuyao-system-controller   openfuyao-system-controller-798c4f6598-npmjs         1/1     Running            0             29m
openfuyao-system              application-management-service-77457c5c85-g7b6z      2/2     Running            0             28m
openfuyao-system              console-service-7fdb88c9c6-g2kp5                     1/1     Running            0             28m
openfuyao-system              console-website-c78945fcc-dm6s4                      1/1     Running            0             28m
openfuyao-system              local-harbor-chartmuseum-57fdd9949d-5xwh6            1/1     Running            0             28m
openfuyao-system              local-harbor-core-6f7d4cc767-9rp6l                   1/1     Running            0             28m
openfuyao-system              local-harbor-database-0                              1/1     Running            0             28m
openfuyao-system              local-harbor-jobservice-7b8c9bf798-ms8qh             1/1     Running            4 (27m ago)   28m
openfuyao-system              local-harbor-nginx-78b94f7b74-nflct                  1/1     Running            0             28m
openfuyao-system              local-harbor-portal-6b8cbf6747-gq6wd                 1/1     Running            0             28m
openfuyao-system              local-harbor-redis-0                                 1/1     Running            0             28m
openfuyao-system              local-harbor-registry-7879c9d46d-d8wq6               2/2     Running            0             28m
openfuyao-system              marketplace-service-5cf7cd6f5b-bmknq                 2/2     Running            0             28m
openfuyao-system              monitoring-service-6fd8dbd59f-bwq96                  2/2     Running            0             28m
openfuyao-system              oauth-server-64f6545c48-2hm7z                        1/1     Running            0             26m
openfuyao-system              oauth-webhook-78cb864fc5-gsmw9                       1/1     Running            0             26m
openfuyao-system              plugin-management-service-6fd8f64cc4-zvql8           2/2     Running            0             26m
openfuyao-system              user-management-operator-9bb7bf64-6q9xg              1/1     Running            0             26m
openfuyao-system              web-terminal-service-6f64b888f9-gdxpb                1/1     Running            0             26m
volcano-system                volcano-controllers-6ffb787f8d-chk69                 0/1     Pending            0             7m58s
volcano-system                volcano-scheduler-867f9784bb-tvhzh                   0/1     Pending            0             7m57s

[root@master1 fuyao-26-03]# kubectl  -n kube-system logs ascend-device-plugin-5hmhf
Defaulted container "device-plugin-01" out of: device-plugin-01, init-permission (init)
[INFO]     2026/03/27 02:26:56.030212 1       hwlog/api.go:108    devicePlugin.log's logger init success
[INFO]     2026/03/27 02:26:56.030715 1       main.go:187    ascend device plugin starting and the version is v6.0.0_linux-aarch64
[INFO]     2026/03/27 02:26:56.030759 1       main.go:188    ascend device plugin starting scene is center
2026/03/27 02:26:56 command exec failed, exit status 1
[ERROR]    2026/03/27 02:26:56.032950 1       devmanager/devmanager.go:95    deviceManager init failed, prepare dcmi failed, err: cannot found valid driver lib, fromEnv: lib path is invalid, [], fromLdCmd: can't find valid lib: EOF
[ERROR]    2026/03/27 02:26:56.033013 1       main.go:203    init devmanager failed, err: auto init failed, err: get chip info failed, err: device Manager is nil, may encounter an exception during initialization. You can check the system log to confirm

环境检查

参考 https://gitcode.com/openFuyao/sig-installation/blob/master/docs/zh/user_guide/cluster_installation_deployment/environment_pre_check_tool_guide.md

wget https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/env-check/releases/download/latest/bin/linux/arm64/envCheck
wget https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/env-check/releases/download/latest/bin/linux/arm64/config.json

文件检查：

[root@localhost env]# ./envCheck query
[INFO][2026-03-27 02:53:25] envCheck tool start
[INFO][2026-03-27 02:53:25] mode: fileQuery
[INFO][2026-03-27 02:53:25] file query start ......
[INFO][2026-03-27 02:53:25] query path: [/root/.kube /etc/kubernetes /usr/local/bin/crictl /etc/sysctl.d/k8s.conf /etc/systemd/system/kubelet.service /etc/systemd/system/kubelet.service.d /var/lib/etcd /var/lib/kubelet /run/containerd/containerd.sock /usr/lib/systemd/system/kubelet.service.d /var/run/containerd/containerd.sock /var/run/docker.sock]
[INFO][2026-03-27 02:53:25] path not exist: /root/.kube
[INFO][2026-03-27 02:53:25] path not exist: /etc/kubernetes
[INFO][2026-03-27 02:53:25] path not exist: /usr/local/bin/crictl
[INFO][2026-03-27 02:53:25] path not exist: /etc/sysctl.d/k8s.conf
[INFO][2026-03-27 02:53:25] path not exist: /etc/systemd/system/kubelet.service
[INFO][2026-03-27 02:53:25] path not exist: /etc/systemd/system/kubelet.service.d
[INFO][2026-03-27 02:53:25] path not exist: /var/lib/etcd
[INFO][2026-03-27 02:53:25] path not exist: /var/lib/kubelet
[INFO][2026-03-27 02:53:25] path not exist: /run/containerd/containerd.sock
[INFO][2026-03-27 02:53:25] path not exist: /usr/lib/systemd/system/kubelet.service.d
[INFO][2026-03-27 02:53:25] path not exist: /var/run/containerd/containerd.sock
[WARNING][2026-03-27 02:53:25] file exist: /var/run/docker.sock
[INFO][2026-03-27 02:53:25] file query completed

Query Time: 2026-03-27 02:53:25

+-------------------------------------------+---------+---------+-------+--------+-------------+
|                   Path                    | Exists  |  Type   | Owner | Group  | Permissions |
+-------------------------------------------+---------+---------+-------+--------+-------------+
|                /root/.kube                | Missing | Missing |-------+--------+-------------+
|              /etc/kubernetes              | Missing | Missing |-------+--------+-------------+
|           /usr/local/bin/crictl           | Missing | Missing |-------+--------+-------------+
|          /etc/sysctl.d/k8s.conf           | Missing | Missing |-------+--------+-------------+
|    /etc/systemd/system/kubelet.service    | Missing | Missing |-------+--------+-------------+
|   /etc/systemd/system/kubelet.service.d   | Missing | Missing |-------+--------+-------------+
|               /var/lib/etcd               | Missing | Missing |-------+--------+-------------+
|             /var/lib/kubelet              | Missing | Missing |-------+--------+-------------+
|      /run/containerd/containerd.sock      | Missing | Missing |-------+--------+-------------+
| /usr/lib/systemd/system/kubelet.service.d | Missing | Missing |-------+--------+-------------+
|    /var/run/containerd/containerd.sock    | Missing | Missing |-------+--------+-------------+
|           /var/run/docker.sock            | Exists  |  File   | root  | docker | Srw-rw----  |
+-------------------------------------------+---------+---------+-------+--------+-------------+

+-------------------+-------+
|      Summary      | Count |
+-------------------+-------+
|   Total Checked   |  12   |
|   Total Exists    |   1   |
|   Total Missing   |  11   |
| Total Directories |   0   |
|    Total Files    |   1   |
+-------------------+-------+

[INFO][2026-03-27 02:53:25] completed

程序存在性检测：

[root@localhost env]# ./envCheck check
[INFO][2026-03-27 02:54:04] envCheck tool start
[INFO][2026-03-27 02:54:04] mode: programCheck
[INFO][2026-03-27 02:54:04] program check start......
[INFO][2026-03-27 02:54:04] os: linux, arch: arm64
[INFO][2026-03-27 02:54:04] program list to check: [docker kubectl containerd]
[INFO][2026-03-27 02:54:04] check program: docker
[WARNING][2026-03-27 02:54:04] docker installed - version: Docker version 18.09.0, build d51e3ad
[INFO][2026-03-27 02:54:04] check program: kubectl
[INFO][2026-03-27 02:54:04] not install: kubectl
[INFO][2026-03-27 02:54:04] check program: containerd
[WARNING][2026-03-27 02:54:04] containerd installed - version: time="2026-03-27T02:54:04Z" level=warning msg="init error, wrong runtimeTimeout format: time: invalid duration """ 
containerd  version:1.2.0.320.oe2203sp4 871075eb7cc979944ba2d987719cb534bbb87e5c
[INFO][2026-03-27 02:54:04] program check completed
[WARNING][2026-03-27 02:54:04] detected installed application(s): docker, containerd. Please uninstall it(them) yourself

Check Time: 2026-03-27 02:54:04

+------------+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+
|  Program   |    Status     |                                                                                               Version                                                                                                |        Path         |
+------------+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+
|   docker   |   Installed   |                                                                                Docker version 18.09.0, build d51e3ad                                                                                 |   /usr/bin/docker   |
|  kubectl   | Not Installed |                                                                                               Unknown                                                                                                |      Not found      |
| containerd |   Installed   | time="2026-03-27T02:54:04Z" level=warning msg="init error, wrong runtimeTimeout format: time: invalid duration """ 
containerd  version:1.2.0.320.oe2203sp4 871075eb7cc979944ba2d987719cb534bbb87e5c | /usr/bin/containerd |
+------------+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+

+-----------------+-------+
|     Summary     | Count |
+-----------------+-------+
|  Total Checked  |   3   |
| Total Installed |   2   |
|  Total Missing  |   1   |
+-----------------+-------+

[INFO][2026-03-27 02:54:04] completed

在线部署

部署完毕浏览 NPU Operator 文档才发现需满足 openEuler 22.03 lts 条件，故重装 https://docs.openfuyao.cn/zh/docs/v25.12/user_guide/npu_operator.html#%E5%AE%89%E8%A3%85

  • CPU: Kunpeng-920
  • OS: openEuler 22.03 (LTS-SP4) aarch64
  • Fuyao Version: v26.03 rc3
  • docker: 2:18.09.0-346.oe2403sp3

使用 openEuler 22.03 (LTS-SP4) aarch64 cloud 镜像全新安装、扩容硬盘后部署。

[root@localhost fuyao-26.3-rc3]# ./bkeadm_linux_arm64 init --otherRepo cr.openfuyao.cn/openfuyao/bke-online-installed:latest
--hostIP:            10.17.30.131
--domain:            deploy.bocloud.k8s
--kubernetesPort:    36443
--imageRepoPort:     40443
--yumRepoPort:       40080
--chartRepoPort:     38080
--ntpServer:         cn.pool.ntp.org:123
--runtime:           containerd
--runtimeStorage:    /var/lib/containerd
--clusterAPI:        1.2.1
--oFVersion:         v26.03-rc.3
--versionUrl:        https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/version-config/
--enableNTP:         true
--agentHealthPort:   58080
--otherRepo:         cr.openfuyao.cn/openfuyao/bke-online-installed:latest
Confirm the parameters, press Y to continue N will exit. [Y/N]? y
[bke][2026-03-27 02:57:16][INFO] BKE initialize ...
[bke][2026-03-27 02:57:16][INFO] HOSTNAME: localhost
[bke][2026-03-27 02:57:16][INFO] PLATFORM: openeuler
[bke][2026-03-27 02:57:16][INFO] Version:  22.03
[bke][2026-03-27 02:57:16][INFO] KERNEL:   5.10.0-216.0.0.115.oe2203sp4.aarch64
[bke][2026-03-27 02:57:16][INFO] GOOS:     linux
[bke][2026-03-27 02:57:16][INFO] ARCH:     arm64
[bke][2026-03-27 02:57:16][INFO] CPU:      8
[bke][2026-03-27 02:57:16][INFO] MEMORY:   16G
[bke][2026-03-27 02:57:16][INFO] BKE Console: ENABLED
[bke][2026-03-27 02:57:16][INFO] BKE initialize environment check...
[bke][2026-03-27 02:57:16][WARN] Client authentication enabled but no credentials provided
2026-03-27T02:57:16.485Z        info    infrastructure/infrastructure.go:53     The docker client is ready.
[bke][2026-03-27 02:57:16][INFO] set up the host machine zone
[bke][2026-03-27 02:57:16][INFO] set ntp server
Fri Mar 27 02:57:15 AM CST 2026

[bke][2026-03-26 18:57:15][INFO] config local source
[bke][2026-03-26 18:57:15][INFO] 在线模式：domain：deploy.bocloud.k8s 绑定到默认IP：10.17.30.131
[bke][2026-03-26 18:57:49][INFO] iptables -V output: iptables v1.8.7 (legacy)
[bke][2026-03-26 18:57:49][INFO] workDir /bke mod permission success
[bke][2026-03-26 18:57:49][ERROR] Failed to start the container service, open /bke/mount/source_registry/files: no such file or directory

最新版似乎修改了命令参数，使用最新参数正常。

[root@stl-fuyao-2603 fuyao-26.3-rc3]# ./bkeadm_linux_arm64 init --onlineImage cr.openfuyao.cn/openfuyao/bke-online-installed:latest
--hostIP:            10.17.30.131
--domain:            deploy.bocloud.k8s
--kubernetesPort:    36443
--imageRepoPort:     40443
--yumRepoPort:       40080
--chartRepoPort:     38080
--ntpServer:         cn.pool.ntp.org:123
--runtime:           containerd
--runtimeStorage:    /var/lib/containerd
--clusterAPI:        1.2.1
--oFVersion:         v26.03-rc.3
--versionUrl:        https://openfuyao.obs.cn-north-4.myhuaweicloud.com/openFuyao/version-config/
--enableNTP:         true
--agentHealthPort:   58080
--onlineImage:       cr.openfuyao.cn/openfuyao/bke-online-installed:latest
Confirm the parameters, press Y to continue N will exit. [Y/N]? y
[bke][2026-03-27 11:36:51][INFO] BKE initialize ...
[bke][2026-03-27 11:36:51][INFO] HOSTNAME: stl-fuyao-2603
[bke][2026-03-27 11:36:51][INFO] PLATFORM: openeuler
[bke][2026-03-27 11:36:51][INFO] Version:  22.03
[bke][2026-03-27 11:36:51][INFO] KERNEL:   5.10.0-216.0.0.115.oe2203sp4.aarch64
[bke][2026-03-27 11:36:51][INFO] GOOS:     linux
[bke][2026-03-27 11:36:51][INFO] ARCH:     arm64
[bke][2026-03-27 11:36:51][INFO] CPU:      8
[bke][2026-03-27 11:36:51][INFO] MEMORY:   16G
[bke][2026-03-27 11:36:51][INFO] BKE Console: ENABLED
[bke][2026-03-27 11:36:51][INFO] BKE initialize environment check...
[bke][2026-03-27 11:36:51][WARN] Client authentication enabled but no credentials provided
[bke][2026-03-27 11:36:51][INFO] set up the host machine zone
[bke][2026-03-27 11:36:51][INFO] set ntp server
Fri Mar 27 11:36:51 AM CST 2026

[bke][2026-03-27 11:36:51][INFO] config local source
[bke][2026-03-27 11:36:51][INFO] 在线模式：domain：deploy.bocloud.k8s 绑定到默认IP：10.17.30.131
[bke][2026-03-27 11:36:51][WARN] 无法解析私有仓库地址，跳过CA证书配置
[bke][2026-03-27 11:36:51][INFO] Using client certificate authentication(CA only)
[bke][2026-03-27 11:36:51][INFO] Download source file...
[bke][2026-03-27 11:36:53][INFO] Writing data...
...
[bke][2026-03-27 13:48:19][INFO] containerd sandbox image: hub.oepkgs.net/openfuyao/pause:3.9
[bke][2026-03-27 13:48:19][INFO] Created offline special hosts.toml: /etc/containerd/certs.d/127.0.0.1:40443/hosts.toml
[bke][2026-03-27 13:48:19][INFO] wait for containerd to start
[bke][2026-03-27 13:48:19][INFO] Waiting for containerd to be ready
...
成功解压： /opt/cni/bin/vrf , 共处理了 4007138 个字符
[bke][2026-03-27 13:48:20][INFO] Start the base dependency service
[bke][2026-03-27 13:48:21][INFO] Image hub.oepkgs.net/openfuyao/registry:2.8.1 is downloading
[bke][2026-03-27 13:48:29][INFO] Wait for the container mirroring service to start...
[bke][2026-03-27 13:48:34][INFO] The container mirroring service is started by containerd. 
[bke][2026-03-27 13:48:34][INFO] Image hub.oepkgs.net/openfuyao/nginx:1.23.0-alpine is downloading
[bke][2026-03-27 13:48:43][INFO] Wait for the container yum service to start...
[bke][2026-03-27 13:48:48][INFO] The container yum service is started. 
[bke][2026-03-27 13:48:48][INFO] Image hub.oepkgs.net/openfuyao/helm/chartmuseum:v0.16.2 is downloading
[bke][2026-03-27 13:49:05][INFO] Wait for the chart mirroring service to start...
[bke][2026-03-27 13:49:10][INFO] The chart mirroring service is started. 
[bke][2026-03-27 13:49:10][INFO] Image hub.oepkgs.net/openfuyao/openebs/nfs-server-alpine:0.9.0 is downloading
[bke][2026-03-27 13:49:17][INFO] Wait for the nfs mirroring service to start...
[bke][2026-03-27 13:49:22][INFO] The nfs mirroring service is started. 
[bke][2026-03-27 13:49:22][INFO] Image hub.oepkgs.net/openfuyao/rancher/k3s:v1.25.16-k3s4 is downloading
2026-03-27T13:50:07.914+0800    info    k3s/k3s.go:314  params: onlineImage=cr.openfuyao.cn/openfuyao/bke-online-installed:latest otherRepo=, otherRepoIp=, hostIP=10.17.30.131, imageRepo=deploy.bocloud.k8s, imageRepoPort=40443, kubernetesPort=36443
[bke][2026-03-27 13:50:07][INFO] Start the local Kubernetes cluster...
[bke][2026-03-27 13:50:10][ERROR] Failed to copy kubectl from the container
[bke][2026-03-27 13:50:10][ERROR] Failed to start kubernetes exit status 1
[bke][2026-03-27 13:50:10][ERROR] Failed to start cluster API, exit status 1

似乎启动失败，居然是因为 tar 没有安装？

[root@stl-fuyao-2603 fuyao-26.3-rc3]# nerdctl cp kubernetes:/bin/k3s /tmp/test-k3s
FATA[0000] unable to copy: failed to find `tar` binary

安装 tar 后解决，顺利部署。

Npu Operator

NAMESPACE                     NAME                                                          READY   STATUS             RESTARTS        AGE
cluster-system                bkeagent-deployer-h2flc                                       1/1     Running            0               65m
ingress-nginx                 ingress-nginx-admission-create-xqwmf                          0/1     Completed          0               62m
ingress-nginx                 ingress-nginx-admission-patch-4dcjr                           0/1     Completed          1               62m
ingress-nginx                 ingress-nginx-controller-p5nrm                                1/1     Running            0               62m
kube-system                   ascend-device-plugin-xtfkb                                    0/1     CrashLoopBackOff   7 (4m34s ago)   16m
kube-system                   calico-kube-controllers-6d75d78f5d-tfjvf                      1/1     Running            0               65m
kube-system                   calico-node-xtd99                                             1/1     Running            0               65m
kube-system                   coredns-6c6fdbdb66-24cn4                                      1/1     Running            0               65m
kube-system                   coredns-6c6fdbdb66-w29mt                                      1/1     Running            1 (63m ago)     65m
kube-system                   etcd-master1                                                  1/1     Running            0               64m
kube-system                   kube-apiserver-master1                                        1/1     Running            0               55m
kube-system                   kube-controller-manager-master1                               1/1     Running            1 (55m ago)     65m
kube-system                   kube-proxy-xxgz4                                              1/1     Running            0               65m
kube-system                   kube-scheduler-master1                                        1/1     Running            1 (55m ago)     65m
kube-system                   metrics-server-586f979f47-4b6fz                               1/1     Running            0               54m
mindx-dl                      ascend-operator-manager-5d4d89f675-tbjpv                      0/1     Pending            0               15m
mindx-dl                      clusterd-5588c5dc88-2fd5g                                     1/1     Running            0               16m
mindx-dl                      resilience-controller-8686bbd76f-jv5lp                        1/1     Running            0               16m
monitoring                    alertmanager-main-0                                           2/2     Running            0               51m
monitoring                    alertmanager-main-1                                           2/2     Running            0               51m
monitoring                    alertmanager-main-2                                           2/2     Running            0               51m
monitoring                    blackbox-exporter-6d6fbbfc96-cl6sg                            3/3     Running            0               56m
monitoring                    kube-state-metrics-677558db89-9rtbl                           3/3     Running            0               56m
monitoring                    node-exporter-4mzzl                                           2/2     Running            0               56m
monitoring                    prometheus-k8s-0                                              2/2     Running            0               51m
monitoring                    prometheus-k8s-1                                              2/2     Running            0               51m
monitoring                    prometheus-operator-5cb64c846d-8m55t                          2/2     Running            0               56m
npu                           ascend-runtime-containerd-pg5q4                               1/1     Running            0               16m
npu                           npu-driver-66dqf                                              0/1     Init:0/1           0               103s
npu                           npu-feature-discovery-rr68r                                   1/1     Running            0               17m
npu                           npu-operator-5858d99c89-lgd24                                 1/1     Running            0               17m
npu                           npu-operator-node-feature-discovery-gc-5cf8bc768d-6w4b4       1/1     Running            0               17m
npu                           npu-operator-node-feature-discovery-master-5985b5cfcd-swdqs   1/1     Running            0               17m
npu                           npu-operator-node-feature-discovery-worker-q5ct6              1/1     Running            0               17m
openfuyao-system-controller   modify-manifests-master1-sjxrv                                0/1     Completed          0               55m
openfuyao-system-controller   openfuyao-system-controller-8444679b95-jv8jg                  1/1     Running            0               65m
openfuyao-system              application-management-service-75799d4dd6-8vm9w               2/2     Running            0               55m
openfuyao-system              console-service-84bbd85575-grtwx                              1/1     Running            0               55m
openfuyao-system              console-website-855c9d8f65-8btkl                              1/1     Running            0               61m
openfuyao-system              local-harbor-chartmuseum-7f96745849-d7vzk                     1/1     Running            0               56m
openfuyao-system              local-harbor-core-5f847798b8-khkf4                            1/1     Running            1 (51m ago)     56m
openfuyao-system              local-harbor-database-0                                       1/1     Running            0               56m
openfuyao-system              local-harbor-jobservice-7d67f4f887-svklq                      1/1     Running            3 (47m ago)     56m
openfuyao-system              local-harbor-nginx-6449749746-pbzkc                           1/1     Running            0               56m
openfuyao-system              local-harbor-portal-78bf65c9-rqmk6                            1/1     Running            0               56m
openfuyao-system              local-harbor-redis-0                                          1/1     Running            0               56m
openfuyao-system              local-harbor-registry-65884895bf-ff2wd                        2/2     Running            0               56m
openfuyao-system              marketplace-service-5c79cbcbfc-swg9r                          2/2     Running            0               55m
openfuyao-system              monitoring-service-79fc57c6b4-47vbm                           2/2     Running            0               56m
openfuyao-system              oauth-server-68b6655d95-c8pzx                                 1/1     Running            0               54m
openfuyao-system              oauth-webhook-6995d46758-g47xn                                1/1     Running            0               54m
openfuyao-system              plugin-management-service-84bfcd6565-2bmz2                    2/2     Running            0               54m
openfuyao-system              user-management-operator-8d79bd8b8-jr4sk                      1/1     Running            0               54m
openfuyao-system              web-terminal-service-6d858d974-hpjw6                          1/1     Running            0               54m
volcano-system                volcano-controllers-6ffb787f8d-mxxph                          1/1     Running            0               16m
volcano-system                volcano-scheduler-867f9784bb-vsswv                            0/1     Pending            0               16m

kube-system                   ascend-device-plugin-sbxr7                           0/1     CrashLoopBackOff   6 (100s ago)   9m57s
[root@master1 ~]# kubectl -n kube-system logs ascend-device-plugin-sbxr7  
Defaulted container "device-plugin-01" out of: device-plugin-01, init-permission (init)  
[INFO]     2026/03/27 07:27:47.189809 1       hwlog/api.go:108    devicePlugin.log's logger init success  
[INFO]     2026/03/27 07:27:47.190348 1       main.go:187    ascend device plugin starting and the version is v6.0.0_linux-aarch64  
[INFO]     2026/03/27 07:27:47.190416 1       main.go:188    ascend device plugin starting scene is center  
2026/03/27 07:27:47 command exec failed, exit status 1  
[ERROR]    2026/03/27 07:27:47.192892 1       devmanager/devmanager.go:95    deviceManager init failed, prepare dcmi failed, err: cannot found valid driver lib, fromEnv: lib path is invalid, [], fromLdCmd: can't find valid lib: EOF  
[ERROR]    2026/03/27 07:27:47.192970 1       main.go:203    init devmanager failed, err: auto init failed, err: get chip info failed, err: device Manager is nil, may encounter an exception during initialization. You can check the system log to confirm

驱动安装失败：

[root@master1 ~]# kubectl -n npu logs -f npu-driver-lxv5c -c npu-driver-installer
Checking if /mnt/usr/local/sbin/npu-smi exists...
master1 is not an option, please use -h to view help
[2026-03-27 15:55:00, [INFO] No operation specified, default install operation on node: --
[2026-03-27 15:55:00, [INFO] install npu-driver
[2026-03-27 15:55:00, [INFO] copy npu-install to host
[2026-03-27 15:55:00, [INFO] copy npu-install to host success
[2026-03-27 15:55:00,303677070] [INFO] Install dependency packages
[2026-03-27 15:55:00,305026980] [INFO] Using yum for package installation
5 files removed
repo                                            4.3 MB/s |  33 kB     00:00    
Metadata cache created.
[2026-03-27 15:55:01,143913450] [INFO] jq is already installed, skipping...
[2026-03-27 15:55:01,150749780] [INFO] wget is already installed, skipping...
[2026-03-27 15:55:01,156894190] [INFO] Installing unzip...
No match for argument: unzip
Error: Unable to find a match: unzip
[2026-03-27 15:55:01,599984660] [FATAL] Failed to install unzip using yum
[2026-03-27 15:55:01, [FATAL] install failed: --

因为主机缺少 unizp 导致失败，但是该容器没有任何异常，无法感知错误。 从清单可以看到 ascend-device-plugin-sbxr7 容器一直由于找不到驱动异常退出，而实际原因是 npu-driver-66dqf pod 没有正常安装驱动，但没有退出，反而正常运行。模糊了实际错误点。

安装 unzip 后正常：

[root@master1 ~]# kubectl -n npu logs -f npu-driver-66dqf -c npu-driver-installer
Checking if /mnt/usr/local/sbin/npu-smi exists...
master1 is not an option, please use -h to view help
[2026-03-27 15:57:58, [INFO] No operation specified, default install operation on node: --
[2026-03-27 15:57:58, [INFO] install npu-driver
[2026-03-27 15:57:58, [INFO] copy npu-install to host
[2026-03-27 15:57:58, [INFO] copy npu-install to host success
[2026-03-27 15:57:58,069791170] [INFO] Install dependency packages
[2026-03-27 15:57:58,070929430] [INFO] Using yum for package installation
48 files removed
repo                                            5.1 MB/s |  33 kB     00:00    
OS                                               67 MB/s | 3.3 MB     00:00    
everything                                       76 MB/s |  17 MB     00:00    
EPOL                                             65 MB/s | 4.7 MB     00:00    
debuginfo                                        69 MB/s | 3.9 MB     00:00    
source                                           59 MB/s | 1.8 MB     00:00    
update                                           79 MB/s |  71 MB     00:00

可以看到花费了大量时间在 init 阶段，这样才是正常的：

npu                           npu-driver-66dqf                                              0/1     Init:0/1           0             3m51s

虚拟机运行似乎会有这个错误

kubectl  -n kube-system delete pod ascend-device-plugin-xtfkb

[Driver] [2026-03-27 16:03:40] [INFO]upgradePercentage:100%
[Driver] [2026-03-27 16:03:42] [INFO]Driver package installed successfully! The new version takes effect immediately.
[Driver] [2026-03-27 16:03:42] [INFO]End time: 2026-03-27 16:03:42
[Firmware] [2026-03-27 16:03:43] [INFO]Start time: 2026-03-27 16:03:43
[Firmware] [2026-03-27 16:03:43] [INFO]LogFile: /var/log/ascend_seclog/ascend_install.log
[Firmware] [2026-03-27 16:03:43] [INFO]OperationLogFile: /var/log/ascend_seclog/operation.log
[Firmware] [2026-03-27 16:03:43] [WARNING]Do not power off or restart the system during the installation/upgrade
[Firmware] [2026-03-27 16:03:43] [ERROR]Not a physical-machine, firmware upgrade does not support.
[Firmware] [2026-03-27 16:03:43] [INFO]End time: 2026-03-27 16:03:43

但驱动等已经部署完毕：

[root@master1 ~]# npu-smi info  
+--------------------------------------------------------------------------------------------------------+  
| npu-smi 24.1.rc3                                 Version: 24.1.rc3                                     |  
+-------------------------------+-----------------+------------------------------------------------------+  
| NPU     Name                  | Health          | Power(W)     Temp(C)           Hugepages-Usage(page) |  
| Chip    Device                | Bus-Id          | AICore(%)    Memory-Usage(MB)                        |  
+===============================+=================+======================================================+  
| 176     310P3                 | OK              | NA           56                0     / 0             |  
| 0       0                     | 0000:00:16.0    | 0            1838 / 21527                            |  
+===============================+=================+======================================================+  
+-------------------------------+-----------------+------------------------------------------------------+  
| NPU     Chip                  | Process id      | Process name             | Process memory(MB)        |  
+===============================+=================+======================================================+  
| No running processes found in NPU 176                                                                  |  
+===============================+=================+======================================================+

但是这个容器依然无法正常运行：

[root@master1 ~]# kubectl -n kube-system logs -f -l name=ascend-device-plugin-ds -c device-plugin-01
[INFO]     2026/03/27 08:19:41.677826 1       hwlog/api.go:108    devicePlugin.log's logger init success
[INFO]     2026/03/27 08:19:41.678051 1       main.go:187    ascend device plugin starting and the version is v6.0.0_linux-aarch64
[INFO]     2026/03/27 08:19:41.678116 1       main.go:188    ascend device plugin starting scene is center
[INFO]     2026/03/27 08:19:41.900653 1       devmanager/devmanager.go:104    the dcmi version is 24.1.rc3
[ERROR]    2026/03/27 08:19:41.900744 1       devmanager/devmanager.go:211    get error card quantity: 0
[ERROR]    2026/03/27 08:19:41.900780 1       devmanager/devmanager.go:195    get card list failed for init
[ERROR]    2026/03/27 08:19:41.900828 1       main.go:203    init devmanager failed, err: auto init failed, err: get card list failed for init

部署后防火墙问题

另外发现 openEuler iptables 默认，还是 openFuyao 默认配置，iptables 默认 FORWARD 为 Drop，会导致部署后无法访问。

[root@stl-fuyao-2603 ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
CNI-ISOLATION-STAGE-1  all  --  0.0.0.0/0            0.0.0.0/0            /* CNI firewall plugin rules (ingressPolicy: same-bridge) */
CNI-FORWARD  all  --  0.0.0.0/0            0.0.0.0/0            /* CNI firewall plugin rules */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

需 配置解决

iptables -P FORWARD ACCEPT

部署时软件源问题

[root@master1 ~]# cp /etc/yum.repos.d/b
bak/      bke.repo

默认移除系统自带软件源，部署后无法再安装软件，需自行配置。 能否通过更加灵活的方式，如配置优先级的方式来规避直接移除软件源。

压缩工具检查

建议参考这种：

root@hosthatch-us1:~# sudo -v ; curl https://rclone.org/install.sh | sudo bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current               
                                 Dload  Upload   Total   Spent    Left  Speed
100  4734  100  4734    0     0   8458      0 --:--:-- --:--:-- --:--:--  8468

None of the supported tools for extracting zip archives (unzip 7z busybox) were found. Please install one of them and try again.

带有明确错误说明。

openFuyao InferNex AI推理集成部署 310P(300I Pro) 环境问题记录及解决

发表评论

AI推理集成部署（InferNex）是一个专为云原生环境下AI推理服务优化所设计的端到端集成部署方案。该方案基于Kubernetes Gateway API Inference Extension (GIE) 和主流LLM技术栈构建，通过Helm Chart将开源网关、智能路由、高性能推理后端、全局KVCache管理、扩缩容决策框架及推理可观测体系等核心加速模块无缝集成。它提供从请求接入、动态路由、推理执行到资源管理与监控的完整加速链路，旨在提升推理吞吐量并降低TTFT/TPOT时延，实现一站式的高效AI服务部署体验。

相关的文档如下：

因为官方仅针对 910 做了验证，手头只有一张 310P ，理论上是可以跑起来，但是需要做一系列修改，本文记录部署遇到的各种问题及其解决方案。

部署后有几个 pod 一直起不来：

NAMESPACE                     NAME                                                          READY   STATUS      RESTARTS       AGE  
ai-inference                  vllm-pd-2p1d-01-decode-54cc4c7579-5h62w                       0/1     Pending     0              5d18h  
ai-inference                  vllm-pd-2p1d-01-decode-6cd64bc69c-7slqg                       0/1     Init:0/2    0              5d  
ai-inference                  vllm-pd-2p1d-01-prefill-5c546dbcc-thmkd                       0/1     Pending     0              5d18h  
ai-inference                  vllm-pd-2p1d-01-prefill-fd68f87cf-jjdlc                       0/1     Pending     0              5d

hccn 问题

似乎是 hccn 找不到

[root@master1 ~]# kubectl  -n ai-inference describe pod vllm-pd-2p1d-01-decode-6cd64bc69c-7slqg 
Name:             vllm-pd-2p1d-01-decode-6cd64bc69c-7slqg
Namespace:        ai-inference
Priority:         0
Service Account:  default
Node:             master1/10.17.30.131
Start Time:       Tue, 07 Apr 2026 09:31:33 +0800
Labels:           app.kubernetes.io/instance=infernex-vllm-pd-2p1d-01
                  app.kubernetes.io/name=inference-backend
                  openfuyao.com/dpSize=1
                  openfuyao.com/engine=vllm
                  openfuyao.com/model=qwen-qwen3-8b
                  openfuyao.com/pdGroupID=qwen3-8b-pd-01
                  openfuyao.com/pdRole=decode
                  openfuyao.com/ppSize=1
                  openfuyao.com/tpSize=1
                  pod-template-hash=6cd64bc69c
Annotations:      checksum/config: 476b32f01fc96ff2896aee7fce288cd2b58cdb2ac825d1a22518798806847a2c
                  huawei.com/AscendReal: Ascend310P-0
                  huawei.com/kltDev: Ascend310P-0
Status:           Pending
IP:               
IPs:              
<none>
Controlled By:    ReplicaSet/vllm-pd-2p1d-01-decode-6cd64bc69c
Init Containers:
  mooncake-config-init:
    Container ID:  
    Image:         hub.oepkgs.net/openfuyao/mikefarah/yq:4.50.1
    Image ID:      
    Port:          
<none>
    Host Port:     
<none>
    Command:
      /bin/sh
      -c
    Args:
      set -e
      CONFIG_PATH="/app/mooncake.json"
      mkdir -p "$(dirname "$CONFIG_PATH")"
      cat > /tmp/mooncake_config.tpl << 'EOF'
        local_hostname: "$POD_IP"
        metadata_server: "redis://redis-service:6379"
        master_server_address: "mooncake-master-service:30089"
        device_name: ""
        protocol: "ascend"
        global_segment_size: 42949672960
        use_ascend_direct: true

      EOF
      POD_IP_VALUE="${POD_IP:-0.0.0.0}"
      sed "s/\$POD_IP/${POD_IP_VALUE}/g" /tmp/mooncake_config.tpl | yq eval - -o=json > "$CONFIG_PATH"

    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      POD_NAME:  vllm-pd-2p1d-01-decode-6cd64bc69c-7slqg (v1:metadata.name)
      POD_IP:     (v1:status.podIP)
    Mounts:
      /app from mooncake-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jswfw (ro)
  huggingface-download:
    Container ID:  
    Image:         cr.openfuyao.cn/openfuyao/huggingface-download:0.22.2
    Image ID:      
    Port:          
<none>
    Host Port:     
<none>
    Command:
      hf
      download
      Qwen/Qwen3-8B
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      HF_HUB_OFFLINE:        0
      VLLM_USE_V1:           1
      GLOO_SOCKET_IFNAME:    eth0
      TP_SOCKET_IFNAME:      eth0
      HCCL_SOCKET_IFNAME:    eth0
      MOONCAKE_CONFIG_PATH:  /app/mooncake.json
    Mounts:
      /root/.cache from rootcache (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jswfw (ro)
Containers:
  decode-engine:
    Container ID:  
    Image:         hub.oepkgs.net/openfuyao/ascend/vllm-ascend:v0.13.0
    Image ID:      
    Port:          8000/TCP (decode-port)
    Host Port:     0/TCP (decode-port)
    Command:
      /bin/bash
      -c
    Args:
      # PHYSICAL_DEVICES stands for the physical devices assigned to the container, use for vllm ascend 0.10.x
      export PHYSICAL_DEVICES=$(ls /dev/davinci* 2>/dev/null | grep -o '[0-9]\+' | sort -n | paste -sd',' -)

      # start vllm service
      vllm serve Qwen/Qwen3-8B \
        --served-model-name Qwen/Qwen3-8B \
        --trust-remote-code \
        --no-enable-prefix-caching \
        --port 8000 \
        --tensor-parallel-size 1 \
        --max-model-len 10000 \
        --max-num-batched-tokens 40960 \
        --data-parallel-size 1 \
        --pipeline-parallel-size 1 \
        --gpu-memory-utilization 0.8 \
        --kv-transfer-config '{"engine_id":"'$POD_NAME'","kv_connector":"MultiConnector","kv_connector_extra_config":{"connectors":[{"kv_buffer_device":"npu","kv_connector":"MooncakeConnectorV1","kv_connector_extra_config":{"decode":{"dp_size":1,"tp_size":1},"prefill":{"dp_size":1,"tp_size":2},"use_ascend_direct":true},"kv_parallel_size":1,"kv_port":"20001","kv_role":"kv_consumer"},{"kv_buffer_device":"npu","kv_connector":"AscendStoreConnector","kv_connector_extra_config":{"backend":"mooncake","decode":{"dp_size":1,"tp_size":1},"lookup_rpc_port":"0","prefill":{"dp_size":1,"tp_size":2}},"kv_parallel_size":1,"kv_port":"20001","kv_role":"kv_consumer"}],"decode":{"dp_size":1,"tp_size":1},"prefill":{"dp_size":1,"tp_size":2}},"kv_port":"20001","kv_rank":1,"kv_role":"kv_consumer"}'

    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:                    8
      huawei.com/Ascend310P:  1
      memory:                 64Gi
    Requests:
      cpu:                    4
      huawei.com/Ascend310P:  1
      memory:                 32Gi
    Liveness:                 http-get http://:decode-port/health delay=0s timeout=10s period=10s #success=1 #failure=3
    Readiness:                http-get http://:decode-port/v1/models delay=0s timeout=5s period=10s #success=1 #failure=3
    Startup:                  http-get http://:decode-port/v1/models delay=30s timeout=5s period=30s #success=1 #failure=60
    Environment:
      HF_HUB_OFFLINE:        0
      VLLM_USE_V1:           1
      GLOO_SOCKET_IFNAME:    eth0
      TP_SOCKET_IFNAME:      eth0
      HCCL_SOCKET_IFNAME:    eth0
      MOONCAKE_CONFIG_PATH:  /app/mooncake.json
      POD_NAME:              vllm-pd-2p1d-01-decode-6cd64bc69c-7slqg (v1:metadata.name)
      POD_IP:                 (v1:status.podIP)
    Mounts:
      /app from mooncake-config (ro)
      /dev/shm from shm (rw)
      /etc/ascend_install.info from installinfo (rw)
      /etc/hccn.conf from hccnconf (rw)
      /root/.cache from rootcache (rw)
      /usr/bin/hccn_tool from hccntool (rw)
      /usr/local/Ascend/driver/lib64 from lib64 (rw)
      /usr/local/Ascend/driver/version.info from version (rw)
      /usr/local/bin/npu-smi from npusmi (rw)
      /usr/local/dcmi from dcmi (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jswfw (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False 
  Initialized                 False 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  mooncake-config:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  
<unset>
  shm:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  24Gi
  dcmi:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/dcmi
    HostPathType:  
  npusmi:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/bin/npu-smi
    HostPathType:  File
  lib64:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/Ascend/driver/lib64
    HostPathType:  
  version:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/Ascend/driver/version.info
    HostPathType:  File
  installinfo:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ascend_install.info
    HostPathType:  File
  hccntool:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/bin/hccn_tool
    HostPathType:  File
  hccnconf:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/hccn.conf
    HostPathType:  File
  rootcache:
    Type:          HostPath (bare host directory volume)
    Path:          /home/llm_cache
    HostPathType:  
  kube-api-access-jswfw:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              
<none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 30s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 30s
Events:
  Type     Reason            Age                     From               Message
  ----     ------            ----                    ----               -------
  Warning  FailedScheduling  5d                      default-scheduler  0/1 nodes are available: 1 Insufficient memory. no new claims to deallocate, preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
  Warning  FailedScheduling  5d (x2 over 5d)         default-scheduler  0/1 nodes are available: 1 Insufficient memory. no new claims to deallocate, preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
  Warning  FailedScheduling  9m32s                   default-scheduler  0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  9m20s (x24 over 9m29s)  default-scheduler  0/1 nodes are available: 1 Insufficient huawei.com/Ascend310P. no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal   Scheduled         8m58s                   default-scheduler  Successfully assigned ai-inference/vllm-pd-2p1d-01-decode-6cd64bc69c-7slqg to master1
  Warning  FailedMount       44s (x12 over 8m58s)    kubelet            MountVolume.SetUp failed for volume "hccntool" : hostPath type check failed: /usr/bin/hccn_tool is not a file

暂时绕过：

touch /usr/bin/hccn_tool
chmod +x /usr/bin/hccn_tool

huggingface-download 失败

[root@master1 ~]# kubectl  -n ai-inference describe pod vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp 
Name:             vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp
Namespace:        ai-inference
Priority:         0
Service Account:  default
Node:             master1/10.17.30.131
Start Time:       Tue, 07 Apr 2026 09:45:19 +0800
Labels:           app.kubernetes.io/instance=infernex-vllm-pd-2p1d-01
                  app.kubernetes.io/name=inference-backend
                  openfuyao.com/dpSize=1
                  openfuyao.com/engine=vllm
                  openfuyao.com/model=qwen-qwen3-8b
                  openfuyao.com/pdGroupID=qwen3-8b-pd-01
                  openfuyao.com/pdRole=decode
                  openfuyao.com/ppSize=1
                  openfuyao.com/tpSize=1
                  pod-template-hash=6cd64bc69c
Annotations:      checksum/config: 476b32f01fc96ff2896aee7fce288cd2b58cdb2ac825d1a22518798806847a2c
                  cni.projectcalico.org/containerID: 32b3384131b69054ca45acc8afe5e272b0ca681ea6d0611b3fec7316e3532e80
                  cni.projectcalico.org/podIP: 192.168.137.155/32
                  cni.projectcalico.org/podIPs: 192.168.137.155/32
                  huawei.com/AscendReal: Ascend310P-0
                  huawei.com/kltDev: Ascend310P-0
Status:           Pending
IP:               192.168.137.155
IPs:
  IP:           192.168.137.155
Controlled By:  ReplicaSet/vllm-pd-2p1d-01-decode-6cd64bc69c
Init Containers:
  mooncake-config-init:
    Container ID:  containerd://4ede488dd17e33f3a980aee6fa4eac3093ad6366c3854fe85436f81f6e1df7bb
    Image:         hub.oepkgs.net/openfuyao/mikefarah/yq:4.50.1
    Image ID:      hub.oepkgs.net/openfuyao/mikefarah/yq@sha256:4facc66fdcc785ec961ef7f2185f53f862f462eefe1d50c2eb311c2bb26823e3
    Port:          
<none>
    Host Port:     
<none>
    Command:
      /bin/sh
      -c
    Args:
      set -e
      CONFIG_PATH="/app/mooncake.json"
      mkdir -p "$(dirname "$CONFIG_PATH")"
      cat > /tmp/mooncake_config.tpl << 'EOF'
        local_hostname: "$POD_IP"
        metadata_server: "redis://redis-service:6379"
        master_server_address: "mooncake-master-service:30089"
        device_name: ""
        protocol: "ascend"
        global_segment_size: 42949672960
        use_ascend_direct: true

      EOF
      POD_IP_VALUE="${POD_IP:-0.0.0.0}"
      sed "s/\$POD_IP/${POD_IP_VALUE}/g" /tmp/mooncake_config.tpl | yq eval - -o=json > "$CONFIG_PATH"

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 07 Apr 2026 09:45:20 +0800
      Finished:     Tue, 07 Apr 2026 09:45:20 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:  vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp (v1:metadata.name)
      POD_IP:     (v1:status.podIP)
    Mounts:
      /app from mooncake-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9jvbd (ro)
  huggingface-download:
    Container ID:  containerd://84e15b7d9e2f7382181e309c1558174ec48e58ad0ae14f92ae0dfff284da76e5
    Image:         cr.openfuyao.cn/openfuyao/huggingface-download:0.22.2
    Image ID:      cr.openfuyao.cn/openfuyao/huggingface-download@sha256:ac86348b5e6934a020c21c4f0ebf81b520194ba8e549f1847ecc7521b82d9a8d
    Port:          
<none>
    Host Port:     
<none>
    Command:
      hf
      download
      Qwen/Qwen3-8B
    State:          Running
      Started:      Tue, 07 Apr 2026 09:47:33 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 07 Apr 2026 09:45:20 +0800
      Finished:     Tue, 07 Apr 2026 09:47:32 +0800
    Ready:          False
    Restart Count:  1
    Environment:
      HF_HUB_OFFLINE:        0
      VLLM_USE_V1:           1
      GLOO_SOCKET_IFNAME:    eth0
      TP_SOCKET_IFNAME:      eth0
      HCCL_SOCKET_IFNAME:    eth0
      MOONCAKE_CONFIG_PATH:  /app/mooncake.json
    Mounts:
      /root/.cache from rootcache (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9jvbd (ro)
Containers:
  decode-engine:
    Container ID:  
    Image:         hub.oepkgs.net/openfuyao/ascend/vllm-ascend:v0.13.0
    Image ID:      
    Port:          8000/TCP (decode-port)
    Host Port:     0/TCP (decode-port)
    Command:
      /bin/bash
      -c
    Args:
      # PHYSICAL_DEVICES stands for the physical devices assigned to the container, use for vllm ascend 0.10.x
      export PHYSICAL_DEVICES=$(ls /dev/davinci* 2>/dev/null | grep -o '[0-9]\+' | sort -n | paste -sd',' -)

      # start vllm service
      vllm serve Qwen/Qwen3-8B \
        --served-model-name Qwen/Qwen3-8B \
        --trust-remote-code \
        --no-enable-prefix-caching \
        --port 8000 \
        --tensor-parallel-size 1 \
        --max-model-len 10000 \
        --max-num-batched-tokens 40960 \
        --data-parallel-size 1 \
        --pipeline-parallel-size 1 \
        --gpu-memory-utilization 0.8 \
        --kv-transfer-config '{"engine_id":"'$POD_NAME'","kv_connector":"MultiConnector","kv_connector_extra_config":{"connectors":[{"kv_buffer_device":"npu","kv_connector":"MooncakeConnectorV1","kv_connector_extra_config":{"decode":{"dp_size":1,"tp_size":1},"prefill":{"dp_size":1,"tp_size":2},"use_ascend_direct":true},"kv_parallel_size":1,"kv_port":"20001","kv_role":"kv_consumer"},{"kv_buffer_device":"npu","kv_connector":"AscendStoreConnector","kv_connector_extra_config":{"backend":"mooncake","decode":{"dp_size":1,"tp_size":1},"lookup_rpc_port":"0","prefill":{"dp_size":1,"tp_size":2}},"kv_parallel_size":1,"kv_port":"20001","kv_role":"kv_consumer"}],"decode":{"dp_size":1,"tp_size":1},"prefill":{"dp_size":1,"tp_size":2}},"kv_port":"20001","kv_rank":1,"kv_role":"kv_consumer"}'

    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:                    8
      huawei.com/Ascend310P:  1
      memory:                 64Gi
    Requests:
      cpu:                    4
      huawei.com/Ascend310P:  1
      memory:                 32Gi
    Liveness:                 http-get http://:decode-port/health delay=0s timeout=10s period=10s #success=1 #failure=3
    Readiness:                http-get http://:decode-port/v1/models delay=0s timeout=5s period=10s #success=1 #failure=3
    Startup:                  http-get http://:decode-port/v1/models delay=30s timeout=5s period=30s #success=1 #failure=60
    Environment:
      HF_HUB_OFFLINE:        0
      VLLM_USE_V1:           1
      GLOO_SOCKET_IFNAME:    eth0
      TP_SOCKET_IFNAME:      eth0
      HCCL_SOCKET_IFNAME:    eth0
      MOONCAKE_CONFIG_PATH:  /app/mooncake.json
      POD_NAME:              vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp (v1:metadata.name)
      POD_IP:                 (v1:status.podIP)
    Mounts:
      /app from mooncake-config (ro)
      /dev/shm from shm (rw)
      /etc/ascend_install.info from installinfo (rw)
      /etc/hccn.conf from hccnconf (rw)
      /root/.cache from rootcache (rw)
      /usr/bin/hccn_tool from hccntool (rw)
      /usr/local/Ascend/driver/lib64 from lib64 (rw)
      /usr/local/Ascend/driver/version.info from version (rw)
      /usr/local/bin/npu-smi from npusmi (rw)
      /usr/local/dcmi from dcmi (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9jvbd (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 False 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  mooncake-config:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  
<unset>
  shm:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  24Gi
  dcmi:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/dcmi
    HostPathType:  
  npusmi:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/bin/npu-smi
    HostPathType:  File
  lib64:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/Ascend/driver/lib64
    HostPathType:  
  version:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/Ascend/driver/version.info
    HostPathType:  File
  installinfo:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ascend_install.info
    HostPathType:  File
  hccntool:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/bin/hccn_tool
    HostPathType:  File
  hccnconf:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/hccn.conf
    HostPathType:  File
  rootcache:
    Type:          HostPath (bare host directory volume)
    Path:          /home/llm_cache
    HostPathType:  
  kube-api-access-9jvbd:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              
<none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 30s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 30s
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  3m20s               default-scheduler  0/1 nodes are available: 1 Insufficient huawei.com/Ascend310P. no new claims to deallocate, preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
  Normal   Scheduled         2m15s               default-scheduler  Successfully assigned ai-inference/vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp to master1
  Normal   Pulled            2m14s               kubelet            Container image "hub.oepkgs.net/openfuyao/mikefarah/yq:4.50.1" already present on machine
  Normal   Created           2m14s               kubelet            Created container: mooncake-config-init
  Normal   Started           2m14s               kubelet            Started container mooncake-config-init
  Normal   Pulled            1s (x2 over 2m14s)  kubelet            Container image "cr.openfuyao.cn/openfuyao/huggingface-download:0.22.2" already present on machine
  Normal   Created           1s (x2 over 2m14s)  kubelet            Created container: huggingface-download
  Normal   Started           1s (x2 over 2m14s)  kubelet            Started container huggingface-download

查看错误日志：

# 看当前这次的日志
kubectl -n ai-inference logs vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp -c huggingface-download

# 看上一次失败的日志
kubectl -n ai-inference logs vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp -c huggingface-download --previous

[root@master1 ~]# kubectl -n ai-inference logs vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp -c huggingface-download
[root@master1 ~]# 
[root@master1 ~]# kubectl -n ai-inference logs vllm-pd-2p1d-01-decode-6cd64bc69c-rq9tp -c huggingface-download --previous
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 250, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    raise exc
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 78, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 207, in connect_tcp
    with map_exceptions(exc_map):
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 240, in snapshot_download
    repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 3285, in repo_info
    return method(
           ^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 3020, in model_info
    r = get_session().get(path, headers=headers, timeout=timeout, params=params)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1053, in get
    return self.request(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/hf", line 8, in 
<module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/cli/hf.py", line 113, in main
    app()
  File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 1152, in __call__
    raise e
  File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 1135, in __call__
    return get_command(self)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/typer/core.py", line 795, in main
    return _main(
           ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/typer/core.py", line 188, in _main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/typer/main.py", line 1514, in wrapper
    return callback(**use_params)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/cli/download.py", line 224, in download
    _print_result(run_download())
                  ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/cli/download.py", line 185, in run_download
    return snapshot_download(
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 324, in snapshot_download
    raise LocalEntryNotFoundError(
huggingface_hub.errors.LocalEntryNotFoundError: Got: ConnectError: [Errno 101] Network is unreachable
An error happened while trying to locate the files on the Hub and we cannot find the appropriate snapshot folder for the specified revision on the local disk. Please check your internet connection and try again.

确认是网络问题：节点无法访问 HuggingFace（Network is unreachable），而且本地也没有缓存。 解决方案：换国内镜像源（推荐） 在 Deployment 的 huggingface-download init container 里加一个环境变量：

env:
  - name: HF_ENDPOINT
    value: "https://hf-mirror.com"

最好给 decode-engine 也加上，否则报同样的错。

看日志可能没有任何输出

kubectl -n ai-inference logs deployments/vllm-pd-2p1d-01-decode huggingface-download  -f

此时查看 llm 目录大小即可，可以看到不断在变化：

$ watch -n 2 -d 'du -sh /home/llm_cache/'
426M    /home/llm_cache/

310p 运行报错

[root@master1 ~]# kubectl  -n ai-inference logs  vllm-pd-2p1d-01-decode-7d487c49cd-qw89v
Defaulted container "decode-engine" out of: decode-engine, mooncake-config-init (init), huggingface-download (init)
...
INFO 04-07 05:15:19 [__init__.py:217] Platform plugin ascend is activated
(EngineCore_DP0 pid=94) INFO 04-07 05:15:33 [ascend_config.py:55] Linear layer sharding enabled with config: None. Note: This feature works optimally with FLASHCOMM2 and DSA-CP enabled; using it without these features may result in significant performance degradation.
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68] EngineCore failed to start.
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68] Traceback (most recent call last):
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_core.py", line 59, in run_engine_core
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     super().__init__(
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 102, in __init__
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     self._init_executor()
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm/vllm/v1/executor/uniproc_executor.py", line 46, in _init_executor
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     self.driver_worker.init_worker(all_kwargs=[kwargs])
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm/vllm/v1/worker/worker_base.py", line 313, in init_worker
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     self.worker = worker_class(**kwargs)
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]                   ^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker.py", line 116, in __init__
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     check_ascend_device_type()
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]   File "/vllm-workspace/vllm-ascend/vllm_ascend/utils.py", line 708, in check_ascend_device_type
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]     assert _ascend_device_type == cur_device_type, f"Current device type: {cur_device_type} does not match the installed version's device type: {_ascend_device_type}, please check your installation package."
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94) ERROR 04-07 05:15:33 [patch_core.py:68] AssertionError: Current device type: AscendDeviceType._310P does not match the installed version's device type: AscendDeviceType.A2, please check your installation package.
(EngineCore_DP0 pid=94) Process EngineCore_DP0:
(EngineCore_DP0 pid=94) Traceback (most recent call last):
(EngineCore_DP0 pid=94)   File "/usr/local/python3.11.14/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=94)     self.run()
(EngineCore_DP0 pid=94)   File "/usr/local/python3.11.14/lib/python3.11/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=94)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_core.py", line 72, in run_engine_core
(EngineCore_DP0 pid=94)     raise e
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_core.py", line 59, in run_engine_core
(EngineCore_DP0 pid=94)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=94)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=94)     super().__init__(
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 102, in __init__
(EngineCore_DP0 pid=94)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=94)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=94)     self._init_executor()
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm/vllm/v1/executor/uniproc_executor.py", line 46, in _init_executor
(EngineCore_DP0 pid=94)     self.driver_worker.init_worker(all_kwargs=[kwargs])
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm/vllm/v1/worker/worker_base.py", line 313, in init_worker
(EngineCore_DP0 pid=94)     self.worker = worker_class(**kwargs)
(EngineCore_DP0 pid=94)                   ^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker.py", line 116, in __init__
(EngineCore_DP0 pid=94)     check_ascend_device_type()
(EngineCore_DP0 pid=94)   File "/vllm-workspace/vllm-ascend/vllm_ascend/utils.py", line 708, in check_ascend_device_type
(EngineCore_DP0 pid=94)     assert _ascend_device_type == cur_device_type, f"Current device type: {cur_device_type} does not match the installed version's device type: {_ascend_device_type}, please check your installation package."
(EngineCore_DP0 pid=94)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=94) AssertionError: Current device type: AscendDeviceType._310P does not match the installed version's device type: AscendDeviceType.A2, please check your installation package.

经过排查是 hub.oepkgs.net/openfuyao/ascend/vllm-ascend:v0.13.0 镜像是针对 910 构建的，找到了官方说明

310p 应该使用带有 310p 后缀的镜像。在 镜像仓库 寻找后替换为 quay.io/ascend/vllm-ascend:v0.18.0rc1-310p-openeuler 尝试。

对比 sha256 发现 hub.oepkgs.net/openfuyao/ascend/vllm-ascend:v0.13.0 镜像是完全的 quay.io/ascend/vllm-ascend:v0.13.0 镜像，sha256 完全一致。

310p vllm-ascend 报错

但是替换为其他带有 310p 后缀的 image ，启动后会报错：

Every 1.0s: kubectl -n ai-inference get pod                                                            master1: Thu Apr  9 08:36:06 2026

NAME                                          READY   STATUS             RESTARTS        AGE
cache-indexer-deployment-65d5b449f6-x9l46     1/1     Running            0               17h
inference-gateway-istio-5f9b7d78f6-7kbrw      1/1     Running            26 (15h ago)    17h
infernex-epp-5cc456bd-4vvmv                   1/1     Running            0               17h
mooncake-master-deployment-74cc5666b7-fr4fq   1/1     Running            0               17h
redis-server-deployment-67566b9765-m66lc      1/1     Running            0               17h
vllm-pd-2p1d-01-decode-7687ccb7b-vg98n        0/1     CrashLoopBackOff   161 (31s ago)   16h
vllm-pd-2p1d-01-prefill-66f7564d7f-tdd84      0/1     Pending            0               43h
vllm-pd-2p1d-01-prefill-fd68f87cf-mhtdk       0/1     Pending            0               40h
vllm-pd-2p1d-01-proxy-7ff4f59865-h8xbw        1/1     Running            0               17h

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
(APIServer pid=1)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/kv_transfer/ascend_multi_connector.py", line 5, in 
<module>
(APIServer pid=1)     from vllm_ascend.distributed.kv_transfer.kv_p2p.mooncake_layerwise_connector import MooncakeLayerwiseConnector
(APIServer pid=1)   File "/vllm-workspace/vllm-ascend/vllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_layerwise_connector.py", line 25, in 
<module>
(APIServer pid=1)     from mooncake.engine import TransferEngine  # type: ignore
(APIServer pid=1)     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) ModuleNotFoundError: No module named 'mooncake'
(APIServer pid=1) [ERROR] 2026-04-08-08:58:03 (PID:1, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
(APIServer pid=1) sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
[root@master1 ~]# kubectl  -n ai-inference describe pod vllm-pd-2p1d-01-decode-7687ccb7b-vg98n | grep -i image:
    Image:         hub.oepkgs.net/openfuyao/mikefarah/yq:4.50.1
    Image:         cr.openfuyao.cn/openfuyao/huggingface-download:0.22.2
    Image:         quay.io/ascend/vllm-ascend:main-310p

针对该问题，openfuyao 给出的方案如下：

@tl.s InferNex在310P环境部署问题排查：

vllm-ascend的310P镜像没有加入mooncake，所以在prefill/decode之间kvcache数据传输无法支持。
https://github.com/vllm-project/vllm-ascend/blob/main/Dockerfile.310p

建议使用聚合模式部署，可参考InferNex聚合模式示例，部署时将`inference-backend.services[0].kvTransferConfig` 配置项删除，即可不使用mooncake相关能力：
https://gitcode.com/openFuyao/InferNex/blob/0.22.2/examples/vllm-aggregated-random-values.yaml

vllm-ascend针对310P在线推理文档：
https://docs.vllm.ai/projects/ascend/en/latest/tutorials/hardwares/310p.html#online-inference-on-npu

310P的推理还未验证过，可以尝试v0.13.0或者 v0.18.0rc1，这两个版本vllm官方有文档支持。

aggregated 模式卡个数错误

环境中只有一张 310p，但是默认申请两张，需要修改两处，分别是资源申请个数 1 ，以及 vllm 启动参数 tensor_parallel_size 设定为 1 。

resources:
          limits:
            cpu: "8"
            huawei.com/Ascend310P: "1"
            memory: 64Gi
          requests:
            cpu: "4"
            huawei.com/Ascend310P: "1"
            memory: 32Gi
...
  # start vllm service
          vllm serve Qwen/Qwen3-8B \
            --served-model-name Qwen/Qwen3-8B \
            --trust-remote-code \
            --enable-prefix-caching \
            --port 8000 \
            --tensor-parallel-size 1 \

bf16 数据类型报错

(EngineCore pid=35) [PID: 35] 2026-04-09-08:27:33.217.113 AclNN_Parameter_Error(EZ1001): Tensor self not implemented for DT_BFLOAT16, should be in dtype support list [DT_FLOAT,DT_FLOAT16,DT_INT8,DT_INT16,DT_INT32,DT_INT64,DT_UINT8,DT_BOOL,DT_DOUBLE,].
(EngineCore pid=35) 
(APIServer pid=1) Traceback (most recent call last):
(APIServer pid=1)   File "/usr/local/python3.11.14/bin/vllm", line 6, in 
<module>
(APIServer pid=1)     sys.exit(main())
(APIServer pid=1)              ^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/main.py", line 75, in main
(APIServer pid=1)     args.dispatch_function(args)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/serve.py", line 118, in cmd
(APIServer pid=1)     uvloop.run(run_server(args))
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/uvloop/__init__.py", line 92, in run
(APIServer pid=1)     return runner.run(wrapper())
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/asyncio/runners.py", line 118, in run
(APIServer pid=1)     return self._loop.run_until_complete(task)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=1)     return await main
(APIServer pid=1)            ^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 656, in run_server
(APIServer pid=1)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 670, in run_server_worker
(APIServer pid=1)     async with build_async_engine_client(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 103, in build_async_engine_client
(APIServer pid=1)     async with build_async_engine_client_from_engine_args(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 144, in build_async_engine_client_from_engine_args
(APIServer pid=1)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 225, in from_vllm_config
(APIServer pid=1)     return cls(
(APIServer pid=1)            ^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 154, in __init__
(APIServer pid=1)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 128, in make_async_mp_client
(APIServer pid=1)     return AsyncMPClient(*client_args)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 924, in __init__
(APIServer pid=1)     super().__init__(
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 583, in __init__
(APIServer pid=1)     with launch_core_engines(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 144, in __exit__
(APIServer pid=1)     next(self.gen)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 972, in launch_core_engines
(APIServer pid=1)     wait_for_engine_startup(
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 1031, in wait_for_engine_startup
(APIServer pid=1)     raise RuntimeError(
(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(APIServer pid=1) [ERROR] 2026-04-09-08:27:49 (PID:1, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
(APIServer pid=1) sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute

根据 AI 说法，暂未查证：Ascend 310P 芯片不支持 bfloat16（bf16）数据类型，但 vLLM 在初始化 rotary embedding（RoPE）时使用了 torch.ones(…, dtype=torch.bfloat16)，导致 ACL 算子报错。 310P 的算子库支持的浮点类型只有 float32 和 float16，不包含 bf16。

通过参数强制使用 float16 --dtype half 解决：

bashvllm serve Qwen/Qwen3-8B \
  --dtype half \   # 强制使用 float16 而非 bfloat16
  ...其他参数

npu_dynamic_quant 算子报错

解决以上问题后 pod 可以运行更久

(EngineCore pid=35) INFO 04-09 08:43:11 [weight_utils.py:574] Time spent downloading weights for Qwen/Qwen3-8B: 1.077223 seconds
Loading safetensors checkpoint shards:   0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  20% Completed | 1/5 [00:08<00:32,  8.07s/it]
Loading safetensors checkpoint shards:  40% Completed | 2/5 [00:16<00:24,  8.33s/it]

但最终依然 error ，跟踪报错信息如下：

(EngineCore pid=35) INFO 04-09 08:47:55 [default_loader.py:384] Loading weights took 35.12 seconds
(EngineCore pid=35) INFO 04-09 08:47:57 [model_runner_v1.py:2589] Loading model weights took 17.6043 GB
.(EngineCore pid=35) INFO 04-09 08:48:11 [backends.py:988] Using cache directory: /root/.cache/vllm/torch_compile_cache/4de24ceb58/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=35) INFO 04-09 08:48:11 [backends.py:1048] Dynamo bytecode transform time: 13.07 s
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] EngineCore failed to start.
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] Traceback (most recent call last):
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1073, in run_engine_core
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 839, in __init__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     super().__init__(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 122, in __init__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 245, in _initialize_kv_caches
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 136, in determine_available_memory
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self.collective_rpc("determine_available_memory")
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/executor/uniproc_executor.py", line 78, in collective_rpc
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/serial_utils.py", line 459, in run_method
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/worker_310p.py", line 69, in determine_available_memory
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     self.model_runner.profile_run()
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2550, in profile_run
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     super().profile_run()
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/worker/gpu_model_runner.py", line 5516, in profile_run
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                                         ^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/model_runner_310p.py", line 170, in _dummy_run
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return super()._dummy_run(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2492, in _dummy_run
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     outputs = self._model_forward(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]               ^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1818, in _model_forward
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     hidden_states = self.model(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                     ^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self._call_impl(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return forward_call(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3.py", line 322, in forward
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     hidden_states = self.model(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                     ^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 597, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     output = TorchCompileWithNoGuardsWrapper.__call__(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/compilation/wrapper.py", line 182, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self._call_with_optional_nvtx_range(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/compilation/wrapper.py", line 76, in _call_with_optional_nvtx_range
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return callable_fn(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 845, in compile_wrapper
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     raise e.remove_dynamo_frames() from None  # see TORCHDYNAMO_VERBOSE=1
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 2196, in _call_user_compiler
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     raise BackendCompilerFailed(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 2171, in _call_user_compiler
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     compiled_fn = compiler_fn(gm, example_inputs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/repro/after_dynamo.py", line 156, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     compiled_gm = compiler_fn(gm, example_inputs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/__init__.py", line 2437, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self.compiler_fn(model_, inputs_, **self.kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 81, in inner
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwds)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/compilation/backends.py", line 1063, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     self.configure_post_pass()
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm/vllm/compilation/backends.py", line 847, in configure_post_pass
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     self.pass_manager.configure(self.vllm_config)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/graph_fusion_pass_manager.py", line 55, in configure
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     self.passes.append(AddRMSNormQuantFusionPass(config))
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/passes/norm_quant_fusion_pass.py", line 493, in __init__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     AddRMSNormDynamicQuantPattern(vllm_config, eps=eps).register(self.pattern_match_passes)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/passes/base_pattern.py", line 49, in register
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     pm.register_replacement(pattern_fn, replacement_fn, example_inputs, pm.fwd_only, pm_pass)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_inductor/pattern_matcher.py", line 1552, in register_replacement
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     pattern, gm = gen_pattern_and_search_gm(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 81, in inner
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwds)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_inductor/pattern_matcher.py", line 1760, in gen_pattern_and_search_gm
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     search_gm = trace_fn(search_fn, flat_inputs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_inductor/pattern_matcher.py", line 2115, in fwd_only
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     gm = make_fx(fn, decompositions, tracing_mode="real")(*args)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 2429, in wrapped
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return make_fx_tracer.trace(f, *args)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 2356, in trace
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self._trace_inner(f, *args)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 2318, in _trace_inner
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     t = dispatch_trace(
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]         ^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_compile.py", line 53, in inner
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return disable_fn(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 1044, in _fn
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return fn(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1303, in dispatch_trace
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     graph = tracer.trace(root, concrete_args)  # type: ignore[arg-type]
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 1044, in _fn
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return fn(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/_symbolic_trace.py", line 868, in trace
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     (self.create_arg(fn(*args)),),
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                      ^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1361, in wrapped
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     out = f(*tensors)  # type:ignore[call-arg]
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]           ^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/passes/norm_quant_fusion_pass.py", line 300, in pattern
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     quantized_output = torch.ops.npu.npu_dynamic_quant(out0)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self._op(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1409, in __torch_function__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self._op(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_stats.py", line 28, in wrapper
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return fn(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1534, in __torch_dispatch__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return proxy_call(self, func, self.pre_dispatch, args, kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 994, in proxy_call
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     out = func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]           ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 841, in __call__
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]     return self._op(*args, **kwargs)
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] torch._dynamo.exc.BackendCompilerFailed: backend='<vllm.compilation.backends.VllmBackend object at 0xfffec83c17d0>' raised:
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] RuntimeError: npu_dynamic_quant:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:82 NPU function error: call aclnnDynamicQuantV2 failed, error code is 561103
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] [ERROR] 2026-04-09-08:48:12 (PID:35, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] [PID: 35] 2026-04-09-08:48:12.623.243 AclNN_Parameter_Error(EZ1001): DynamicQuant launch kernel failed.
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]         TraceBack (most recent call last):
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]         Tiling failed
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]         Tiling Failed.
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]         Kernel GetWorkspace failed. opType: 21
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099]         DynamicQuant launch kernel failed.
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] 
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] 
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
(EngineCore pid=35) ERROR 04-09 08:48:12 [core.py:1099] 
(EngineCore pid=35) Process EngineCore:
(EngineCore pid=35) Traceback (most recent call last):
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=35)     self.run()
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/multiprocessing/process.py", line 108, in run
(EngineCore pid=35)     self._target(*self._args, **self._kwargs)
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1103, in run_engine_core
(EngineCore pid=35)     raise e
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1073, in run_engine_core
(EngineCore pid=35)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=35)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 839, in __init__
(EngineCore pid=35)     super().__init__(
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 122, in __init__
(EngineCore pid=35)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=35)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 245, in _initialize_kv_caches
(EngineCore pid=35)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=35)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 136, in determine_available_memory
(EngineCore pid=35)     return self.collective_rpc("determine_available_memory")
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/executor/uniproc_executor.py", line 78, in collective_rpc
(EngineCore pid=35)     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=35)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/serial_utils.py", line 459, in run_method
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/worker_310p.py", line 69, in determine_available_memory
(EngineCore pid=35)     self.model_runner.profile_run()
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2550, in profile_run
(EngineCore pid=35)     super().profile_run()
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/worker/gpu_model_runner.py", line 5516, in profile_run
(EngineCore pid=35)     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=35)                                         ^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/model_runner_310p.py", line 170, in _dummy_run
(EngineCore pid=35)     return super()._dummy_run(
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2492, in _dummy_run
(EngineCore pid=35)     outputs = self._model_forward(
(EngineCore pid=35)               ^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1818, in _model_forward
(EngineCore pid=35)     hidden_states = self.model(
(EngineCore pid=35)                     ^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35)     return self._call_impl(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35)     return forward_call(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3.py", line 322, in forward
(EngineCore pid=35)     hidden_states = self.model(
(EngineCore pid=35)                     ^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 597, in __call__
(EngineCore pid=35)     output = TorchCompileWithNoGuardsWrapper.__call__(
(EngineCore pid=35)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/compilation/wrapper.py", line 182, in __call__
(EngineCore pid=35)     return self._call_with_optional_nvtx_range(
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/compilation/wrapper.py", line 76, in _call_with_optional_nvtx_range
(EngineCore pid=35)     return callable_fn(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 845, in compile_wrapper
(EngineCore pid=35)     raise e.remove_dynamo_frames() from None  # see TORCHDYNAMO_VERBOSE=1
(EngineCore pid=35)     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 2196, in _call_user_compiler
(EngineCore pid=35)     raise BackendCompilerFailed(
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 2171, in _call_user_compiler
(EngineCore pid=35)     compiled_fn = compiler_fn(gm, example_inputs)
(EngineCore pid=35)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/repro/after_dynamo.py", line 156, in __call__
(EngineCore pid=35)     compiled_gm = compiler_fn(gm, example_inputs)
(EngineCore pid=35)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/__init__.py", line 2437, in __call__
(EngineCore pid=35)     return self.compiler_fn(model_, inputs_, **self.kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 81, in inner
(EngineCore pid=35)     return func(*args, **kwds)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/compilation/backends.py", line 1063, in __call__
(EngineCore pid=35)     self.configure_post_pass()
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/compilation/backends.py", line 847, in configure_post_pass
(EngineCore pid=35)     self.pass_manager.configure(self.vllm_config)
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/graph_fusion_pass_manager.py", line 55, in configure
(EngineCore pid=35)     self.passes.append(AddRMSNormQuantFusionPass(config))
(EngineCore pid=35)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/passes/norm_quant_fusion_pass.py", line 493, in __init__
(EngineCore pid=35)     AddRMSNormDynamicQuantPattern(vllm_config, eps=eps).register(self.pattern_match_passes)
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/passes/base_pattern.py", line 49, in register
(EngineCore pid=35)     pm.register_replacement(pattern_fn, replacement_fn, example_inputs, pm.fwd_only, pm_pass)
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_inductor/pattern_matcher.py", line 1552, in register_replacement
(EngineCore pid=35)     pattern, gm = gen_pattern_and_search_gm(
(EngineCore pid=35)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 81, in inner
(EngineCore pid=35)     return func(*args, **kwds)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_inductor/pattern_matcher.py", line 1760, in gen_pattern_and_search_gm
(EngineCore pid=35)     search_gm = trace_fn(search_fn, flat_inputs)
(EngineCore pid=35)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_inductor/pattern_matcher.py", line 2115, in fwd_only
(EngineCore pid=35)     gm = make_fx(fn, decompositions, tracing_mode="real")(*args)
(EngineCore pid=35)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 2429, in wrapped
(EngineCore pid=35)     return make_fx_tracer.trace(f, *args)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 2356, in trace
(EngineCore pid=35)     return self._trace_inner(f, *args)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 2318, in _trace_inner
(EngineCore pid=35)     t = dispatch_trace(
(EngineCore pid=35)         ^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_compile.py", line 53, in inner
(EngineCore pid=35)     return disable_fn(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 1044, in _fn
(EngineCore pid=35)     return fn(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1303, in dispatch_trace
(EngineCore pid=35)     graph = tracer.trace(root, concrete_args)  # type: ignore[arg-type]
(EngineCore pid=35)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 1044, in _fn
(EngineCore pid=35)     return fn(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/_symbolic_trace.py", line 868, in trace
(EngineCore pid=35)     (self.create_arg(fn(*args)),),
(EngineCore pid=35)                      ^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1361, in wrapped
(EngineCore pid=35)     out = f(*tensors)  # type:ignore[call-arg]
(EngineCore pid=35)           ^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/passes/norm_quant_fusion_pass.py", line 300, in pattern
(EngineCore pid=35)     quantized_output = torch.ops.npu.npu_dynamic_quant(out0)
(EngineCore pid=35)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35)     return self._op(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1409, in __torch_function__
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35)     return self._op(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_stats.py", line 28, in wrapper
(EngineCore pid=35)     return fn(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 1534, in __torch_dispatch__
(EngineCore pid=35)     return proxy_call(self, func, self.pre_dispatch, args, kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py", line 994, in proxy_call
(EngineCore pid=35)     out = func(*args, **kwargs)
(EngineCore pid=35)           ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 841, in __call__
(EngineCore pid=35)     return self._op(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) torch._dynamo.exc.BackendCompilerFailed: backend='<vllm.compilation.backends.VllmBackend object at 0xfffec83c17d0>' raised:
(EngineCore pid=35) RuntimeError: npu_dynamic_quant:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:82 NPU function error: call aclnnDynamicQuantV2 failed, error code is 561103
(EngineCore pid=35) [ERROR] 2026-04-09-08:48:12 (PID:35, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
(EngineCore pid=35) [PID: 35] 2026-04-09-08:48:12.623.243 AclNN_Parameter_Error(EZ1001): DynamicQuant launch kernel failed.
(EngineCore pid=35)         TraceBack (most recent call last):
(EngineCore pid=35)         Tiling failed
(EngineCore pid=35)         Tiling Failed.
(EngineCore pid=35)         Kernel GetWorkspace failed. opType: 21
(EngineCore pid=35)         DynamicQuant launch kernel failed.
(EngineCore pid=35) 
(EngineCore pid=35) 
(EngineCore pid=35) Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
(EngineCore pid=35) 
(APIServer pid=1) Traceback (most recent call last):
(APIServer pid=1)   File "/usr/local/python3.11.14/bin/vllm", line 6, in 
<module>
(APIServer pid=1)     sys.exit(main())
(APIServer pid=1)              ^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/main.py", line 75, in main
(APIServer pid=1)     args.dispatch_function(args)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/serve.py", line 118, in cmd
(APIServer pid=1)     uvloop.run(run_server(args))
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/uvloop/__init__.py", line 92, in run
(APIServer pid=1)     return runner.run(wrapper())
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/asyncio/runners.py", line 118, in run
(APIServer pid=1)     return self._loop.run_until_complete(task)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=1)     return await main
(APIServer pid=1)            ^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 656, in run_server
(APIServer pid=1)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 670, in run_server_worker
(APIServer pid=1)     async with build_async_engine_client(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 103, in build_async_engine_client
(APIServer pid=1)     async with build_async_engine_client_from_engine_args(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 144, in build_async_engine_client_from_engine_args
(APIServer pid=1)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 225, in from_vllm_config
(APIServer pid=1)     return cls(
(APIServer pid=1)            ^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 154, in __init__
(APIServer pid=1)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 128, in make_async_mp_client
(APIServer pid=1)     return AsyncMPClient(*client_args)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 924, in __init__
(APIServer pid=1)     super().__init__(
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 583, in __init__
(APIServer pid=1)     with launch_core_engines(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 144, in __exit__
(APIServer pid=1)     next(self.gen)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 972, in launch_core_engines
(APIServer pid=1)     wait_for_engine_startup(
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 1031, in wait_for_engine_startup
(APIServer pid=1)     raise RuntimeError(
(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(APIServer pid=1) [ERROR] 2026-04-09-08:48:30 (PID:1, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
(APIServer pid=1) sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute

根据 ai 的说法：根本原因：norm_quant 融合 Pass 在编译阶段向 310P 发起 npu_dynamic_quant 算子，而 310P 不支持该动态量化算子（或当前 CANN 版本不兼容），导致 Tiling 失败。

openfuyao 社区给出的回应如下：

@tl.s  看了一下报错日志，应该是310卡不支持vllm-ascend默认开启的算子 DynamicQuantV2，可以加上启动配置项 –enforce-eager 和 –no-quant 尝试一下。 在InferNex中，默认未直接提供的vllm启动参数可以在 inference-backend.services[0].pd.prefill/decode.extraArgs添加。例如： extraArgs:

  • “–enforce-eager”
  • “–no-quant “

若还是不行，可以尝试更换模型，按照官方310P文档内的示例部署，如 Qwen2.5-7B-Instruct。 https://docs.vllm.ai/projects/ascend/en/latest/tutorials/hardwares/310p.html#online-inference-on-npu

OOM

[root@master1 fuyao-26.3-rc3]# kubectl  -n ai-inference logs deployments/vllm-pd-2p1d-01 -f 
Defaulted container "aggregated-engine" out of: aggregated-engine, huggingface-download (init)

INFO 04-13 06:21:29 [__init__.py:44] Available plugins for group vllm.platform_plugins:
INFO 04-13 06:21:29 [__init__.py:46] - ascend -> vllm_ascend:register
INFO 04-13 06:21:29 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 04-13 06:21:29 [__init__.py:239] Platform plugin ascend is activated
INFO 04-13 06:21:42 [__init__.py:110] Registered model loader `<class 'vllm_ascend.model_loader.netloader.netloader.ModelNetLoaderElastic'>` with load format `netloader`
INFO 04-13 06:21:42 [__init__.py:110] Registered model loader `<class 'vllm_ascend.model_loader.rfork.rfork_loader.RForkModelLoader'>` with load format `rfork`
WARNING 04-13 06:21:44 [__init__.py:80] The quantization method 'ascend' already exists and will be overwritten by the quantization config <class 'vllm_ascend._310p.quantization.modelslim_config.AscendModelSlimConfig310'>.
(APIServer pid=1) INFO 04-13 06:21:45 [utils.py:297] 
(APIServer pid=1) INFO 04-13 06:21:45 [utils.py:297]        █     █     █▄   ▄█
(APIServer pid=1) INFO 04-13 06:21:45 [utils.py:297]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.18.0
(APIServer pid=1) INFO 04-13 06:21:45 [utils.py:297]   █▄█▀ █     █     █     █  model   Qwen/Qwen2.5-7B-Instruct
(APIServer pid=1) INFO 04-13 06:21:45 [utils.py:297]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1) INFO 04-13 06:21:45 [utils.py:297] 
(APIServer pid=1) INFO 04-13 06:21:45 [utils.py:233] non-default args: {'model_tag': 'Qwen/Qwen2.5-7B-Instruct', 'model': 'Qwen/Qwen2.5-7B-Instruct', 'trust_remote_code': True, 'dtype': 'float16', 'max_model_len': 4096, 'enforce_eager': True, 'served_model_name': ['Qwen/Qwen2.5-7B-Instruct'], 'block_size': 128, 'gpu_memory_utilization': 0.8, 'enable_prefix_caching': True, 'max_num_batched_tokens': 40960, 'kv_events_config': KVEventsConfig(enable_kv_cache_events=True, publisher='zmq', endpoint='tcp://*:5557', replay_endpoint=None, buffer_steps=10000, hwm=100000, max_queue_size=100000, topic='kv-events')}
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_PORT_8000_TCP_ADDR
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_SERVICE_PORT_HTTP_API
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_PORT
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_PORT_8000_TCP_PORT
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_PORT_8000_TCP
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_SERVICE_PORT
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_PORT_8000_TCP_PROTO
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_PD_2P1D_01_SERVICE_HOST
(APIServer pid=1) WARNING 04-13 06:21:45 [envs.py:1717] Unknown vLLM environment variable detected: VLLM_USE_V1
(APIServer pid=1) INFO 04-13 06:22:18 [model.py:533] Resolved architecture: Qwen2ForCausalLM
(APIServer pid=1) WARNING 04-13 06:22:18 [model.py:1920] Casting torch.bfloat16 to torch.float16.
(APIServer pid=1) INFO 04-13 06:22:18 [model.py:1582] Using max model len 4096
(APIServer pid=1) INFO 04-13 06:22:18 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=40960.
(APIServer pid=1) INFO 04-13 06:22:18 [vllm.py:754] Asynchronous scheduling is enabled.
(APIServer pid=1) WARNING 04-13 06:22:18 [vllm.py:788] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(APIServer pid=1) WARNING 04-13 06:22:18 [vllm.py:799] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(APIServer pid=1) INFO 04-13 06:22:18 [vllm.py:964] Cudagraph is disabled under eager mode
(APIServer pid=1) WARNING 04-13 06:22:26 [platform.py:749] Parameter '--disable-cascade-attn' is a GPU-specific feature. Resetting to False for Ascend.
(APIServer pid=1) WARNING 04-13 06:22:26 [platform.py:838] Ignored parameter 'disable_flashinfer_prefill'. This is a GPU-specific feature not supported on Ascend. Resetting to False.
(APIServer pid=1) INFO 04-13 06:22:26 [ascend_config.py:425] Dynamic EPLB is False
(APIServer pid=1) INFO 04-13 06:22:26 [ascend_config.py:426] The number of redundant experts is 0
(APIServer pid=1) INFO 04-13 06:22:26 [platform.py:297] Compilation disabled, using eager mode by default
(APIServer pid=1) INFO 04-13 06:22:26 [platform.py:502] Set PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
(APIServer pid=1) INFO 04-13 06:22:26 [compilation.py:289] Enabled custom fusions: norm_quant, act_quant
INFO 04-13 06:22:53 [__init__.py:44] Available plugins for group vllm.platform_plugins:
INFO 04-13 06:22:53 [__init__.py:46] - ascend -> vllm_ascend:register
INFO 04-13 06:22:53 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 04-13 06:22:53 [__init__.py:239] Platform plugin ascend is activated
(EngineCore pid=35) INFO 04-13 06:23:03 [__init__.py:110] Registered model loader `<class 'vllm_ascend.model_loader.netloader.netloader.ModelNetLoaderElastic'>` with load format `netloader`
(EngineCore pid=35) INFO 04-13 06:23:03 [__init__.py:110] Registered model loader `<class 'vllm_ascend.model_loader.rfork.rfork_loader.RForkModelLoader'>` with load format `rfork`
(EngineCore pid=35) INFO 04-13 06:23:03 [core.py:103] Initializing a V1 LLM engine (v0.18.0) with config: model='Qwen/Qwen2.5-7B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen2.5-7B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=npu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen2.5-7B-Instruct, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'vllm_ascend.compilation.compiler_interface.AscendCompiler', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [40960], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(EngineCore pid=35) WARNING 04-13 06:23:08 [camem.py:66] Failed to import vllm_ascend_C:/vllm-workspace/vllm-ascend/vllm_ascend/vllm_ascend_C.cpython-311-aarch64-linux-gnu.so: undefined symbol: _ZN9pp_matmul17GetPpMatmulTilingERKNS_10MatMulInfoERKNS_12HardwareInfoERjRNS_18PpMatmulTilingDataE. Sleep mode will be disabled. 
(EngineCore pid=35) INFO 04-13 06:23:08 [ascend_config.py:425] Dynamic EPLB is False
(EngineCore pid=35) INFO 04-13 06:23:08 [ascend_config.py:426] The number of redundant experts is 0
INFO 04-13 06:23:22 [__init__.py:44] Available plugins for group vllm.platform_plugins:
INFO 04-13 06:23:22 [__init__.py:46] - ascend -> vllm_ascend:register
INFO 04-13 06:23:22 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 04-13 06:23:22 [__init__.py:239] Platform plugin ascend is activated
....(EngineCore pid=35) INFO 04-13 06:24:35 [parallel_state.py:1395] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.137.164:42089 backend=hccl
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(EngineCore pid=35) INFO 04-13 06:24:36 [parallel_state.py:1717] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(EngineCore pid=35) WARNING 04-13 06:24:36 [worker.py:306] Bind cpus failed in rank0: Can not get running npu info. Skip binding cpu.
(EngineCore pid=35) INFO 04-13 06:24:37 [model_runner_v1.py:2562] Starting to load model Qwen/Qwen2.5-7B-Instruct...
(EngineCore pid=35) INFO 04-13 06:24:56 [weight_utils.py:574] Time spent downloading weights for Qwen/Qwen2.5-7B-Instruct: 4.228922 seconds
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:10<00:30, 10.12s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:20<00:20, 10.32s/it]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:30<00:10, 10.21s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:40<00:00,  9.99s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:40<00:00, 10.08s/it]
(EngineCore pid=35) 
(EngineCore pid=35) INFO 04-13 06:25:46 [default_loader.py:384] Loading weights took 40.53 seconds
(EngineCore pid=35) INFO 04-13 06:25:48 [model_runner_v1.py:2589] Loading model weights took 16.2391 GB
.(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099] EngineCore failed to start.
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099] Traceback (most recent call last):
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1073, in run_engine_core
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 839, in __init__
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     super().__init__(
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 122, in __init__
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 245, in _initialize_kv_caches
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 136, in determine_available_memory
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self.collective_rpc("determine_available_memory")
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/executor/uniproc_executor.py", line 78, in collective_rpc
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/serial_utils.py", line 459, in run_method
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/worker_310p.py", line 69, in determine_available_memory
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     self.model_runner.profile_run()
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2550, in profile_run
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     super().profile_run()
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/v1/worker/gpu_model_runner.py", line 5516, in profile_run
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                                         ^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/model_runner_310p.py", line 170, in _dummy_run
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return super()._dummy_run(
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return func(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2492, in _dummy_run
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     outputs = self._model_forward(
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]               ^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1818, in _model_forward
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     hidden_states = self.model(
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                     ^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self._call_impl(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return forward_call(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 583, in forward
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     hidden_states = self.model(
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                     ^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 439, in __call__
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self.forward(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 444, in forward
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     hidden_states, residual = layer(positions, hidden_states, residual)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self._call_impl(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return forward_call(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 311, in forward
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     hidden_states = self.mlp(hidden_states)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                     ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self._call_impl(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return forward_call(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 114, in forward
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     gate_up, _ = self.gate_up_proj(x)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                  ^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self._call_impl(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return forward_call(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/ops/linear.py", line 215, in forward
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return super().forward(input_)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/layers/linear.py", line 582, in forward
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     output_parallel = self.quant_method.apply(self, input_, bias)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/layers/linear.py", line 228, in apply
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return dispatch_unquantized_gemm()(layer, x, layer.weight, bias)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/worker/patch_unquantized_gemm.py", line 55, in default_unquantized_gemm
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return torch.ops.vllm.unquantized_gemm(x, weight, bias)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self._op(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm/vllm/model_executor/parameter.py", line 126, in __torch_function__
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return super().__torch_function__(func, types, args, kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return self._op(*args, **kwargs)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]   File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/worker/patch_unquantized_gemm.py", line 27, in unquantized_gemm
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]     return torch.nn.functional.linear(x, weight, bias)
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) ERROR 04-13 06:25:51 [core.py:1099] RuntimeError: NPU out of memory. Tried to allocate 2.89 GiB (NPU 0; 21.02 GiB total capacity; 17.36 GiB already allocated; 17.36 GiB current active; 2.15 GiB free; 17.38 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.
(EngineCore pid=35) Process EngineCore:
(EngineCore pid=35) Traceback (most recent call last):
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=35)     self.run()
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/multiprocessing/process.py", line 108, in run
(EngineCore pid=35)     self._target(*self._args, **self._kwargs)
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1103, in run_engine_core
(EngineCore pid=35)     raise e
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1073, in run_engine_core
(EngineCore pid=35)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=35)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 839, in __init__
(EngineCore pid=35)     super().__init__(
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 122, in __init__
(EngineCore pid=35)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=35)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 245, in _initialize_kv_caches
(EngineCore pid=35)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=35)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 136, in determine_available_memory
(EngineCore pid=35)     return self.collective_rpc("determine_available_memory")
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/executor/uniproc_executor.py", line 78, in collective_rpc
(EngineCore pid=35)     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=35)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/serial_utils.py", line 459, in run_method
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/worker_310p.py", line 69, in determine_available_memory
(EngineCore pid=35)     self.model_runner.profile_run()
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2550, in profile_run
(EngineCore pid=35)     super().profile_run()
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/v1/worker/gpu_model_runner.py", line 5516, in profile_run
(EngineCore pid=35)     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=35)                                         ^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/_310p/model_runner_310p.py", line 170, in _dummy_run
(EngineCore pid=35)     return super()._dummy_run(
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore pid=35)     return func(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2492, in _dummy_run
(EngineCore pid=35)     outputs = self._model_forward(
(EngineCore pid=35)               ^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1818, in _model_forward
(EngineCore pid=35)     hidden_states = self.model(
(EngineCore pid=35)                     ^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35)     return self._call_impl(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35)     return forward_call(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 583, in forward
(EngineCore pid=35)     hidden_states = self.model(
(EngineCore pid=35)                     ^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 439, in __call__
(EngineCore pid=35)     return self.forward(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 444, in forward
(EngineCore pid=35)     hidden_states, residual = layer(positions, hidden_states, residual)
(EngineCore pid=35)                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35)     return self._call_impl(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35)     return forward_call(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 311, in forward
(EngineCore pid=35)     hidden_states = self.mlp(hidden_states)
(EngineCore pid=35)                     ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35)     return self._call_impl(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35)     return forward_call(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 114, in forward
(EngineCore pid=35)     gate_up, _ = self.gate_up_proj(x)
(EngineCore pid=35)                  ^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore pid=35)     return self._call_impl(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore pid=35)     return forward_call(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/ops/linear.py", line 215, in forward
(EngineCore pid=35)     return super().forward(input_)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/layers/linear.py", line 582, in forward
(EngineCore pid=35)     output_parallel = self.quant_method.apply(self, input_, bias)
(EngineCore pid=35)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/layers/linear.py", line 228, in apply
(EngineCore pid=35)     return dispatch_unquantized_gemm()(layer, x, layer.weight, bias)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/worker/patch_unquantized_gemm.py", line 55, in default_unquantized_gemm
(EngineCore pid=35)     return torch.ops.vllm.unquantized_gemm(x, weight, bias)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35)     return self._op(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm/vllm/model_executor/parameter.py", line 126, in __torch_function__
(EngineCore pid=35)     return super().__torch_function__(func, types, args, kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore pid=35)     return self._op(*args, **kwargs)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35)   File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/worker/patch_unquantized_gemm.py", line 27, in unquantized_gemm
(EngineCore pid=35)     return torch.nn.functional.linear(x, weight, bias)
(EngineCore pid=35)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=35) RuntimeError: NPU out of memory. Tried to allocate 2.89 GiB (NPU 0; 21.02 GiB total capacity; 17.36 GiB already allocated; 17.36 GiB current active; 2.15 GiB free; 17.38 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.
(APIServer pid=1) Traceback (most recent call last):
(APIServer pid=1)   File "/usr/local/python3.11.14/bin/vllm", line 6, in 
<module>
(APIServer pid=1)     sys.exit(main())
(APIServer pid=1)              ^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/main.py", line 75, in main
(APIServer pid=1)     args.dispatch_function(args)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/serve.py", line 118, in cmd
(APIServer pid=1)     uvloop.run(run_server(args))
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/uvloop/__init__.py", line 92, in run
(APIServer pid=1)     return runner.run(wrapper())
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/asyncio/runners.py", line 118, in run
(APIServer pid=1)     return self._loop.run_until_complete(task)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=1)     return await main
(APIServer pid=1)            ^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 656, in run_server
(APIServer pid=1)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 670, in run_server_worker
(APIServer pid=1)     async with build_async_engine_client(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 103, in build_async_engine_client
(APIServer pid=1)     async with build_async_engine_client_from_engine_args(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 144, in build_async_engine_client_from_engine_args
(APIServer pid=1)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 225, in from_vllm_config
(APIServer pid=1)     return cls(
(APIServer pid=1)            ^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 154, in __init__
(APIServer pid=1)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 128, in make_async_mp_client
(APIServer pid=1)     return AsyncMPClient(*client_args)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 924, in __init__
(APIServer pid=1)     super().__init__(
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 583, in __init__
(APIServer pid=1)     with launch_core_engines(
(APIServer pid=1)   File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 144, in __exit__
(APIServer pid=1)     next(self.gen)
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 972, in launch_core_engines
(APIServer pid=1)     wait_for_engine_startup(
(APIServer pid=1)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 1031, in wait_for_engine_startup
(APIServer pid=1)     raise RuntimeError(
(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(APIServer pid=1) [ERROR] 2026-04-13-06:26:05 (PID:1, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
(APIServer pid=1) sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute

根据官网说明，似乎单张 310P 只能跑个 0.6B

Run the following script to start the vLLM server on NPU (Qwen3-0.6B:1 card, Qwen2.5-7B-Instruct:2 cards, Pangu-Pro-MoE-72B: 8 cards)

https://docs.vllm.ai/projects/ascend/en/latest/tutorials/hardwares/310p.html#online-inference-on-npu

而且需要一些参数

vllm serve Qwen/Qwen3-0.6B \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --enforce-eager \
    --dtype float16

Helm Update 报错

每次更新 helm 都需要经历以下几个回合，才能成功：

[root@master1 fuyao-26.3-rc3]# helm upgrade --install -n ai-inference infernex ./infernex -f infernex.values 
Error: UPGRADE FAILED: cannot patch "vllm-pd-2p1d-01" with kind Deployment: Deployment.apps "vllm-pd-2p1d-01" is invalid: spec.selector: Invalid value: {"matchLabels":{"app.kubernetes.io/instance":"infernex-vllm-pd-2p1d-01","app.kubernetes.io/name":"inference-backend","openfuyao.com/dpSize":"1","openfuyao.com/engine":"vllm","openfuyao.com/model":"qwen-qwen3-0.6b","openfuyao.com/pdRole":"aggregate","openfuyao.com/tpSize":"2"}}: field is immutable
[root@master1 fuyao-26.3-rc3]# kubectl -n ai-inference delete deployments.apps vllm-pd-2p1d-01 
deployment.apps "vllm-pd-2p1d-01" deleted from ai-inference namespace
[root@master1 fuyao-26.3-rc3]# helm upgrade --install -n ai-inference infernex ./infernex -f infernex.values 
Error: UPGRADE FAILED: post-upgrade hooks failed: warning: Hook post-upgrade infernex/charts/pd-orchestrator/charts/resourcescalinggroup/templates/webhook-wait-hook.yaml failed: 1 error occurred:
        * jobs.batch "infernex-resourcescalinggroup-wait-webhook" is forbidden: unable to create new content in namespace scaling-system because it is being terminated

[root@master1 fuyao-26.3-rc3]# helm upgrade --install -n ai-inference infernex ./infernex -f infernex.values 
Error: UPGRADE FAILED: failed to create resource: namespaces "scaling-system" not found
[root@master1 fuyao-26.3-rc3]# helm upgrade --install -n ai-inference infernex ./infernex -f infernex.values 

Release "infernex" has been upgraded. Happy Helming!
NAME: infernex
LAST DEPLOYED: Mon Apr 13 14:33:47 2026
NAMESPACE: ai-inference
STATUS: deployed
REVISION: 27
TEST SUITE: None

istio httproute 错误

[root@master1 fuyao-26.3-rc3]# kubectl  -n ai-inference describe httproutes.gateway.networking.k8s.io qwen-qwen3-0.6b-httproute 
Name:         qwen-qwen3-0.6b-httproute
Namespace:    ai-inference
Labels:       app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=infernex-epp
              app.kubernetes.io/version=0.21.0
Annotations:  meta.helm.sh/release-name: infernex
              meta.helm.sh/release-namespace: ai-inference
API Version:  gateway.networking.k8s.io/v1
Kind:         HTTPRoute
Metadata:
  Creation Timestamp:  2026-04-13T06:31:19Z
  Generation:          1
  Resource Version:    3664436
  UID:                 21dca6e8-483a-4b03-8a78-82559c45a7e3
Spec:
  Parent Refs:
    Group:  gateway.networking.k8s.io
    Kind:   Gateway
    Name:   inference-gateway
  Rules:
    Backend Refs:
      Group:   inference.networking.k8s.io
      Kind:    InferencePool
      Name:    qwen-qwen3-0.6b
      Weight:  1
    Matches:
      Path:
        Type:   PathPrefix
        Value:  /
    Timeouts:
      Request:  300s
Status:
  Parents:
    Conditions:
      Last Transition Time:  2026-04-13T06:31:19Z
      Message:               Route was valid
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2026-04-13T06:31:19Z
      Message:               InferencePool.Name invalid; the name of the InferencePool must be used, not the hostname.
      Observed Generation:   1
      Reason:                InvalidDestination
      Status:                False
      Type:                  ResolvedRefs
    Controller Name:         istio.io/gateway-controller
    Parent Ref:
      Group:  gateway.networking.k8s.io
      Kind:   Gateway
      Name:   inference-gateway
Events:       
<none>

总结

  • helm 默认申请 2 张 310P ，需手动修改 
    resources:
limits:
  cpu: "8"
  huawei.com/Ascend310P: "1"
  memory: 64Gi
requests:
  cpu: "4"
  huawei.com/Ascend310P: "1"
  memory: 32Gi
  • huggerface 下载需手动配置国内源 
    - name: HF_ENDPOINT
  value: https://hf-mirror.com
  • vllm 启动参数需手动调整 
    vllm serve Qwen/Qwen3-0.6B \
--served-model-name Qwen/Qwen3-0.6B \
--trust-remote-code \
--enable-prefix-caching \
--port 8000 \
--tensor-parallel-size 1 \
--max-model-len 4096 \
--enforce-eager \
--dtype float16 \
--max-num-batched-tokens 40960 \
--data-parallel-size 1 \
--gpu-memory-utilization 0.8 \
--block-size 128 \
--kv-events-config '{"enable_kv_cache_events": true, "publisher":"zmq", "topic":"kv-events"}'
  • 默认服务采用 ClusterAPI 使用 Gateway istio 暴露服务，目前还没有正常工作
  • vllm pod 看到频繁打印
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.137.151:51092 - "GET /metrics HTTP/1.1" 200 OK

Refs