<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
    <id>https://project-hami.io/zh/blog</id>
    <title>HAMi 博客</title>
    <updated>2026-11-17T00:00:00.000Z</updated>
    <generator>https://github.com/jpmonette/feed</generator>
    <link rel="alternate" href="https://project-hami.io/zh/blog"/>
    <subtitle>HAMi 项目的最新资讯</subtitle>
    <icon>https://project-hami.io/zh/favicon.ico</icon>
    <rights>Copyright 2026 HAMi Authors</rights>
    <entry>
        <title type="html"><![CDATA[HAMi GPU Descheduler：自动整理 Kubernetes 集群的 GPU 碎片]]></title>
        <id>https://project-hami.io/zh/blog/hami-gpu-descheduler</id>
        <link href="https://project-hami.io/zh/blog/hami-gpu-descheduler"/>
        <updated>2026-11-17T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[GPU 碎片化会让集群 67% 的显存闲置，新 Pod 却一直 Pending。本文介绍碎片化的成因，以及 HAMi GPU Descheduler 如何归整分散的 vGPU 工作负载、找回连续可用的 GPU 容量。]]></summary>
        <content type="html"><![CDATA[<p>GPU 是 Kubernetes 集群里最昂贵的资源，因此大多数团队会选择共享使用。借助 <a href="https://github.com/Project-HAMi/HAMi" target="_blank" rel="noopener noreferrer" class="">HAMi</a>，一张物理 GPU 可以被切分成多个虚拟 GPU 分配给不同的 Pod，每个 Pod 都拥有独立的显存和算力隔离。</p>
<p>但共享也带来了一个长期运行的集群迟早会遇到的问题：<strong>GPU 碎片化</strong>。弹性伸缩、滚动升级、失败重试，Pod 不断创建销毁，每次都会在 GPU 上留下小块碎片。某一天，一个新 Pod 申请一整张 GPU，监控面板显示集群还有大量空闲显存，这个 Pod 却一直 <code>Pending</code>。容量明明存在，只是散落成了无法使用的碎片。</p>
<p>本文解释 GPU 碎片化的成因、手工治理为什么困难，以及 <a href="https://github.com/Project-HAMi/gpu-descheduler" target="_blank" rel="noopener noreferrer" class="">HAMi GPU Descheduler</a> 如何自动化地解决这个问题。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="问题有容量却没有位置">问题：有容量，却没有位置<a href="https://project-hami.io/zh/blog/hami-gpu-descheduler#%E9%97%AE%E9%A2%98%E6%9C%89%E5%AE%B9%E9%87%8F%E5%8D%B4%E6%B2%A1%E6%9C%89%E4%BD%8D%E7%BD%AE" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>考虑一个两节点集群，每个节点通过 HAMi 共享 4 张 GPU，8 个 Pod 各持有小块 vGPU：</p>
<ul>
<li class="">节点 A：<code>a</code> 30%、<code>b</code> 40%、<code>c</code> 25%、<code>d</code> 45%</li>
<li class="">节点 B：<code>e</code> 35%、<code>f</code> 20%、<code>g</code> 40%、<code>h</code> 30%</li>
</ul>
<p>全集群只用了 33% 的显存，67% 空闲。此时一个新 Pod 申请<strong>一整张 GPU</strong>：任何一张卡上都没有完整的 100% 空闲，调度器无法放置，Pod 只能留在 <code>Pending</code> 状态，而你大部分的 GPU 资源正在闲置。</p>
<p><img decoding="async" loading="lazy" alt="GPU 碎片化整理前后对比" src="https://project-hami.io/zh/assets/images/gpu-fragmentation-e6b72329fd339e5f8c654bed01537ba6.svg#light-only" width="1200" height="940" class="img_ev3q"> <img decoding="async" loading="lazy" alt="GPU 碎片化整理前后对比" src="https://project-hami.io/zh/assets/images/gpu-fragmentation-dark-cdbcd27b6d1be92162cf58187307c4ad.svg#dark-only" width="1200" height="940" class="img_ev3q"></p>
<p>这与内存或磁盘的碎片化是同一种现象，区别在于：内存碎片可以靠分配器合并解决，GPU 碎片不能。一个需要整卡的 Pod（例如不做模型并行的推理服务）只能落在一张完全空闲的设备上，分散在不同 GPU 上的碎片无法拼凑成一张整卡。GPU 碎片也并非只挡住整卡申请：任何 vGPU 申请，只要超过了所有设备上最大的单块空闲容量，都会无法落位。比如申请一张 GPU 的 70%，而每张卡的空闲显存都散成 40% 加 30% 的碎块，同样放置不了，整卡申请只是最容易看清的情形。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="碎片化从哪里来">碎片化从哪里来<a href="https://project-hami.io/zh/blog/hami-gpu-descheduler#%E7%A2%8E%E7%89%87%E5%8C%96%E4%BB%8E%E5%93%AA%E9%87%8C%E6%9D%A5" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>碎片化不是 Bug，而是集群动态运行的必然结果：</p>
<ul>
<li class=""><strong>弹性伸缩。</strong> 缩容时删除的 Pod 分散在多个节点上，留下大量半空的 GPU。</li>
<li class=""><strong>滚动升级。</strong> 新版本的 Pod 落在有空余的节点上，旧版本离开后留下空洞。</li>
<li class=""><strong>失败重试。</strong> 训练任务每次重启可能调度到不同节点。</li>
<li class=""><strong>混合负载。</strong> 申请 20% 到 40% 显存的推理 Pod 与需要整卡的任务交错分布。</li>
</ul>
<p>几周之后，集群就会从紧凑布局漂移成散乱布局：总利用率看起来健康，连续的空闲容量却在悄悄消失。更麻烦的是，被碎片挡住的往往正是最重要的任务：小 Pod 总能塞进缝隙，申请份额更大的大模型训练与微调反而最难落位。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="手工治理为什么痛苦">手工治理为什么痛苦<a href="https://project-hami.io/zh/blog/hami-gpu-descheduler#%E6%89%8B%E5%B7%A5%E6%B2%BB%E7%90%86%E4%B8%BA%E4%BB%80%E4%B9%88%E7%97%9B%E8%8B%A6" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>手工治理的思路人尽皆知：找到碎片、<code>kubectl delete pod</code> 删掉可以迁移的 Pod、等控制器重建，期望它们落在合适的位置，必要时用节点选择器固定。反复操作，直到某张卡的空闲容量足以放下等待中的 Pod。</p>
<p>偶尔做一次没问题，但代价是真实的：</p>
<ul>
<li class="">每次驱逐都是一次人工决策，你必须清楚哪些负载可以容忍中断。</li>
<li class="">重建的 Pod 重新走调度器，可能又被摊开，整理成果归零。</li>
<li class="">没有任何意图记录，布局很快再次劣化，同样的操作周而复始。</li>
</ul>
<p>团队真正需要的是一种声明式、一次性的操作：“把这些负载归整到那个节点的 GPU 上”。这正是 HAMi GPU Descheduler 提供的能力。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="hami-gpu-descheduler">HAMi GPU Descheduler<a href="https://project-hami.io/zh/blog/hami-gpu-descheduler#hami-gpu-descheduler" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><a href="https://github.com/Project-HAMi/gpu-descheduler" target="_blank" rel="noopener noreferrer" class="">HAMi GPU Descheduler</a> 是面向 HAMi 虚拟化 Kubernetes 集群的 GPU 碎片整理工具。它通过两个机制归整分散在各节点的 vGPU 工作负载：<strong>Mutating Admission Webhook</strong> 和 <strong><code>HamiPodPlacement</code> CRD</strong>。它扮演的角色与操作系统中的磁盘碎片整理工具一样：把散落的负载归拢到一起，恢复每张设备上连续可用的空闲容量。它与 HAMi Scheduler 互补：调度器负责 GPU 资源的虚拟化、调度与隔离，descheduler 负责把已分配的资源摆放得更紧凑。</p>
<p><img decoding="async" loading="lazy" alt="HAMi GPU Descheduler 归整碎片化 vGPU 工作负载的流程" src="https://project-hami.io/zh/assets/images/how-it-works-9f3b468fe97625cd30f59fb8e3aca450.svg#light-only" width="840" height="640" class="img_ev3q"> <img decoding="async" loading="lazy" alt="HAMi GPU Descheduler 归整碎片化 vGPU 工作负载的流程" src="https://project-hami.io/zh/assets/images/how-it-works-dark-ddbfb01f8aa5c8609b17ce89a1ae4b91.svg#dark-only" width="840" height="640" class="img_ev3q"></p>
<p>整个流程是这样的：</p>
<ol>
<li class="">创建一个 <code>HamiPodPlacement</code>，描述期望的放置：目标节点、GPU 设备、适用的 Pod，可选地指定一个需要驱逐腾位的 Pod。</li>
<li class="">若设置了 <code>evictPod</code>，descheduler 会驱逐该 Pod 并等待；负载所属的控制器（Deployment、StatefulSet 等）将其重建。</li>
<li class="">当目标 Pod（新建的或重建的）进入准入阶段，webhook 将它与 <code>HamiPodPlacement</code> 匹配（<code>reqID</code> 加 <code>podSelector</code>），并注入规划好的放置。</li>
<li class="">Pod 被调度到你指定的节点和 GPU 设备上。每个 <code>HamiPodPlacement</code> 恰好被一个 Pod 消费，保证了操作的幂等与可审计。</li>
</ol>
<p>放置以 CRD 表达，碎片整理于是成为又一个普通的 Kubernetes 操作：声明式、可进 Git 审查、可自动化。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="整理这个集群">整理这个集群<a href="https://project-hami.io/zh/blog/hami-gpu-descheduler#%E6%95%B4%E7%90%86%E8%BF%99%E4%B8%AA%E9%9B%86%E7%BE%A4" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>回到开篇那个两节点集群。要腾空节点 B 并在节点 A 上腾出连续容量，六次一次性放置就够了：Pod <code>e</code>、<code>f</code>、<code>g</code>、<code>h</code> 从节点 B 搬到节点 A，<code>c</code> 和 <code>d</code> 在节点 A 内部重新固定，让 8 个 Pod 归拢到 GPU 0 到 2 上。以 Pod <code>e</code> 为例：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">apiVersion</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> scheduling.hami.placement.com/v1</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">kind</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> HamiPodPlacement</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">metadata</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> consolidate</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">e</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">namespace</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> default</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">spec</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">reqID</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"defrag-e-0001"</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 最短 5 个字符，仅消费一次</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">podSelector</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">matchLabels</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">app</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> inference</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">nodeName</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">node</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">a</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">gpuDevices</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token number" style="color:rgb(247, 140, 108)">1</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 或 gpu-1/70 表示 GPU 1 的 70%</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">evictPod</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> inference</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">e</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">6cd5964877</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">zdb5z </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 需要驱逐并重新放置的 Pod</span><br></div></code></pre></div></div>
<p>关于 <code>podSelector</code> 有一点要注意：它决定哪个 Pod 消费这次放置，而“先通过准入的那个匹配 Pod”会消费它。如果命名空间里所有 Pod 都带着 <code>app: inference</code> 标签，消费放置的可能就不是 <code>e</code>，而是它的某个同门。ReplicaSet 的 <code>pod-template-hash</code> 之类的标签解决不了这个问题：同一个 ReplicaSet 的所有副本都带相同的值，先被准入的同门仍可能消费这次放置。请使用只有被搬迁工作负载才携带的标签，让选择器匹配不到别的 Pod；多副本工作负载则一次只搬一个副本，并避免在有兄弟 Pod 正在创建时进行搬迁。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="evictpod-是什么怎么用"><code>evictPod</code> 是什么、怎么用<a href="https://project-hami.io/zh/blog/hami-gpu-descheduler#evictpod-%E6%98%AF%E4%BB%80%E4%B9%88%E6%80%8E%E4%B9%88%E7%94%A8" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p><code>evictPod</code> 是把“放置”变成“搬迁”的字段。它的值是同一个命名空间里一个现存 Pod 的名字，回答的是一个问题：为了让这次放置成立，哪个正在运行的 Pod 应该让出它现在占用的 GPU？</p>
<p>随后的流程是这样的。descheduler 驱逐这个 Pod 并等待；负载所属的控制器（Deployment、StatefulSet 等）重建该 Pod；重建的 Pod 进入准入阶段，匹配上 <code>reqID</code> 与 <code>podSelector</code>，webhook 把它固定到你规划的 <code>nodeName</code> 和 <code>gpuDevices</code> 上。不设置这个字段时，放置只影响之后创建的 Pod，是一种规划工具；设置之后，你就能主动搬移已经在运行的工作负载，而这正是碎片整理所需要的。</p>
<p>这个字段是可选的，原因也很实际。驱逐发生在目的地校验之前，所以添加之前务必确认 <code>nodeName</code> 和 <code>gpuDevices</code> 指向一个有足够空位的有效目的地，否则 Pod 被驱逐后无处可去。另外，descheduler 只等待替代 Pod、从不自行创建 Pod，因此被驱逐的负载必须由控制器管理。请有意识地使用 <code>evictPod</code>，一次搬一个 Pod。</p>
<p>每个 Pod 被驱逐、重建、并固定到节点 A 的特定 GPU 上，布局逐渐收敛为示意图中“整理后”的状态：8 个 Pod 打包进 3 张 GPU，新 Pod 占用第 4 张，<strong>节点 B 完全腾空</strong>，不增加任何容量，找回 4 张整卡。集群显存利用率从 33% 提升到 43%，提升完全来自那个原本无法启动的新 Pod。</p>
<p>腾空的节点可以缩容降本，也可以转而承接需要整卡的大模型训练任务。</p>
<div class="theme-admonition theme-admonition-warning admonition_xJq3 alert alert--warning"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 16 16"><path fill-rule="evenodd" d="M8.893 1.5c-.183-.31-.52-.5-.887-.5s-.703.19-.886.5L.138 13.499a.98.98 0 0 0 0 1.001c.193.31.53.501.886.501h13.964c.367 0 .704-.19.877-.5a1.03 1.03 0 0 0 .01-1.002L8.893 1.5zm.133 11.497H6.987v-2.003h2.039v2.003zm0-3.004H6.987V5.987h2.039v4.006z"></path></svg></span>驱逐之前</div><div class="admonitionContent_BuS1"><p>主动驱逐会终止正在运行的 Pod，可能短暂降低可用性。负载必须由控制器（Deployment、StatefulSet 等）管理，确保被驱逐的 Pod 会被重建；开始整理前，请检查负载的就绪状态与中断保护。</p></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="快速上手">快速上手<a href="https://project-hami.io/zh/blog/hami-gpu-descheduler#%E5%BF%AB%E9%80%9F%E4%B8%8A%E6%89%8B" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>使用 Helm 部署（要求 HAMi Scheduler v2.10.0+，Kubernetes v1.24+，推荐 v1.28+）：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token function" style="color:rgb(130, 170, 255)">git</span><span class="token plain"> clone https://github.com/Project-HAMi/gpu-descheduler.git</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">cd</span><span class="token plain"> gpu-descheduler</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm </span><span class="token function" style="color:rgb(130, 170, 255)">install</span><span class="token plain"> gpu-descheduler ./chart </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--namespace</span><span class="token plain"> kube-system --create-namespace</span><br></div></code></pre></div></div>
<p>确认 webhook 正常运行：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl get pods </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> kube-system </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-l</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">app</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">gpu-descheduler-webhook </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-w</span><br></div></code></pre></div></div>
<p>webhook 还在 8083 端口提供 REST API，用于查看集群 GPU 状态和放置记录，这是决定整理什么的依据。这个端口没有通过 Service 暴露，需要先把它转发到本机（命令会一直阻塞，请在单独的终端里运行）：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl port-forward </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> kube-system deploy/gpu-descheduler-webhook </span><span class="token number" style="color:rgb(247, 140, 108)">8083</span><span class="token plain">:8083</span><br></div></code></pre></div></div>
<p>然后查询 API：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 查看节点、GPU 设备与健康状态</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token function" style="color:rgb(130, 170, 255)">curl</span><span class="token plain"> </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-s</span><span class="token plain"> http://localhost:8083/api/v1/cluster-info</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 查看当前 Placement CRD 缓存与分配记录</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token function" style="color:rgb(130, 170, 255)">curl</span><span class="token plain"> </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-s</span><span class="token plain"> http://localhost:8083/api/v1/placements</span><br></div></code></pre></div></div>
<p>目标 Pod 必须使用 <code>schedulerName: hami-scheduler</code>，且其标签要匹配 <code>HamiPodPlacement</code> 中的 <code>podSelector</code>。完整流程见<a class="" href="https://project-hami.io/zh/docs/next/userguide/hami-gpu-descheduler">用户指南</a>。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="结语">结语<a href="https://project-hami.io/zh/blog/hami-gpu-descheduler#%E7%BB%93%E8%AF%AD" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>GPU 碎片化是一个随集群动态性增长的可用性与成本问题：容量存在却无法使用，而人工治理脆弱且无据可查。HAMi GPU Descheduler 把碎片整理变成一次声明式的 Kubernetes 操作：驱逐、重建、固定，放置意图以 CRD 表达，由准入 webhook 强制执行。</p>
<p>这一能力源自真实的生产痛点：<a href="https://github.com/Project-HAMi/HAMi/issues/2072" target="_blank" rel="noopener noreferrer" class="">HAMi issue #2072</a> 提出了节点内碎片整理的需求；<a href="https://www.yxqiche.com/" target="_blank" rel="noopener noreferrer" class="">易鑫训推一体平台团队</a>完成了 webhook 加 CRD 方案的设计与主要实现，并将代码开源贡献给 HAMi 社区；仓库公开后，社区维护者和用户持续参与评审与完善。感谢每一位参与其中的社区成员。眼下的整理计划仍由人来制定：把哪些 Pod 归拢到哪个节点，靠的是人工判断，descheduler 负责把计划精确落地，搬迁本身已经可以跨节点进行。社区正在讨论的下一步，是让规划也走向自动化：自动发现集群中的碎片，智能化地生成多个候选治理方案，由用户决策执行哪一个，并把整理的视角从单个节点扩展到整个集群。</p>
<p>如果你在动态集群中用 HAMi 共享 GPU，请留意那些申请较大显存份额却一直 <code>Pending</code> 的 Pod，整卡申请只是其中最容易看清的情形。然后试试 descheduler：</p>
<ul>
<li class=""><strong>代码与发布：</strong> <a href="https://github.com/Project-HAMi/gpu-descheduler" target="_blank" rel="noopener noreferrer" class="">github.com/Project-HAMi/gpu-descheduler</a></li>
<li class=""><strong>用户指南：</strong><a class="" href="https://project-hami.io/zh/docs/next/userguide/hami-gpu-descheduler">使用 HAMi GPU Descheduler 归整 GPU 资源碎片</a></li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="延伸阅读">延伸阅读<a href="https://project-hami.io/zh/blog/hami-gpu-descheduler#%E5%BB%B6%E4%BC%B8%E9%98%85%E8%AF%BB" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class=""><a class="" href="https://project-hami.io/zh/docs/next/userguide/hami-gpu-descheduler">HAMi GPU Descheduler 用户指南</a></li>
<li class=""><a href="https://github.com/Project-HAMi/HAMi/issues/2072" target="_blank" rel="noopener noreferrer" class="">HAMi issue #2072：vGPU 资源节点内碎片化整理</a></li>
<li class=""><a href="https://github.com/Project-HAMi/HAMi" target="_blank" rel="noopener noreferrer" class="">HAMi 项目</a></li>
</ul>]]></content>
        <author>
            <name>宋净超（Jimmy Song）</name>
            <uri>https://github.com/rootsongjc</uri>
        </author>
        <author>
            <name>易鑫训推团队</name>
        </author>
        <category label="HAMi" term="HAMi"/>
        <category label="GPU" term="GPU"/>
        <category label="Kubernetes" term="Kubernetes"/>
        <category label="Descheduler" term="Descheduler"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[HAMi v2.11 Remote GPU 实战：Run CUDA Anywhere]]></title>
        <id>https://project-hami.io/zh/blog/hami-v2-11-remote-gpu</id>
        <link href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu"/>
        <updated>2026-11-12T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[介绍 HAMi v2.11 Remote GPU 的部署方式，在纯 CPU Kubernetes 节点运行 CUDA Samples 和 vLLM，通过远端 T4 完成真实推理。]]></summary>
        <content type="html"><![CDATA[<p>HAMi v2.11.0 新增了 <strong>Remote GPU（远程 GPU 访问）</strong> 功能，可以让 CPU-only 机器访问其他机器上的 GPU，该功能可以让 CUDA 工作负载与 GPU 资源可以跨节点部署。</p>
<p>以往在 Kubernetes 中运行 CUDA 或 vLLM 工作负载，通常需要将 Pod 调度到 GPU 所在的节点。现在，Pod 可以运行在独立的 CPU 节点上，通过网络使用 GPU Server 提供的算力，客户端无需本地 GPU 和 NVIDIA 驱动。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="1-lupine-remote-gpu-介绍">1. Lupine Remote GPU 介绍<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#1-lupine-remote-gpu-%E4%BB%8B%E7%BB%8D" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>Lupine 是一个 <code>GPU-over-IP</code> 项目，可以让 CPU-only 机器通过网络使用其他机器上的 GPU，运行 CUDA 工作负载。</p>
<p>在 Lupine 中节点角色分为 Server 节点和 Client 节点。</p>
<ul>
<li class="">Server 节点拥有 GPU，需要安装 NVIDIA 驱动并运行 Lupine 服务端，负责接收远程 CUDA 调用、执行 GPU 计算并返回结果。</li>
<li class="">Client 节点运行 CUDA 应用，通过加载 Lupine 客户端库，将 CUDA 调用经网络转发给 Server，无需本地 GPU 和 NVIDIA 驱动。</li>
</ul>
<p><img decoding="async" loading="lazy" alt="本地 GPU 与 Remote GPU 的工作原理对比" src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIxMjAwIiBoZWlnaHQ9IjcyMCIgdmlld0JveD0iMCAwIDEyMDAgNzIwIiByb2xlPSJpbWciIGFyaWEtbGFiZWxsZWRieT0idGl0bGUgZGVzY3JpcHRpb24iPgogIDx0aXRsZSBpZD0idGl0bGUiPuacrOWcsCBHUFUg5LiOIFJlbW90ZSBHUFUg55qE5bel5L2c5Y6f55CG5a+55q+UPC90aXRsZT4KICA8ZGVzYyBpZD0iZGVzY3JpcHRpb24iPuacrOWcsOW6lOeUqOmAmui/hyBOVklESUEg6amx5Yqo6K6/6ZeuIEdQVeOAgui/nOeoi+W6lOeUqOWcqCBDUFUgQ2xpZW50IOi/kOihjO+8jOe7jyBMdXBpbmUg5a6i5oi356uv5bqT6L2s5Y+RIENVREEg6LCD55So77yM5ZyoIEdQVSBTZXJ2ZXIg5LiK5a6M5oiQ6K6h566X5bm26L+U5Zue57uT5p6c44CCPC9kZXNjPgogIDxkZWZzPgogICAgPG1hcmtlciBpZD0iYXJyb3ciIG1hcmtlcldpZHRoPSI4IiBtYXJrZXJIZWlnaHQ9IjgiIHJlZlg9IjciIHJlZlk9IjQiIG9yaWVudD0iYXV0byI+CiAgICAgIDxwYXRoIGQ9Ik0gMCAwIEwgOCA0IEwgMCA4IFoiIGZpbGw9IiMzOWM2ZjQiIGZpbGwtb3BhY2l0eT0iMC43NSIvPgogICAgPC9tYXJrZXI+CiAgICA8bWFya2VyIGlkPSJyZXR1cm4tYXJyb3ciIG1hcmtlcldpZHRoPSI4IiBtYXJrZXJIZWlnaHQ9IjgiIHJlZlg9IjciIHJlZlk9IjQiIG9yaWVudD0iYXV0byI+CiAgICAgIDxwYXRoIGQ9Ik0gMCAwIEwgOCA0IEwgMCA4IFoiIGZpbGw9IiMxMGQwNWQiLz4KICAgIDwvbWFya2VyPgogICAgPHN0eWxlPgogICAgICB0ZXh0IHsKICAgICAgICBmaWxsOiAjMWEzOTU2OwogICAgICAgIGZvbnQtZmFtaWx5OiAiU291cmNlIFNhbnMgMyIsICJOb3RvIFNhbnMiLCAiU2Vnb2UgVUkiLCAiUGluZ0ZhbmcgU0MiLCAiTm90byBTYW5zIENKSyBTQyIsICJNaWNyb3NvZnQgWWFIZWkiLCBzYW5zLXNlcmlmOwogICAgICAgIGZvbnQtc2l6ZTogMjBweDsKICAgICAgfQogICAgICAuaGVhZGluZyB7CiAgICAgICAgZmlsbDogIzE1MzI0ZDsKICAgICAgICBmb250LWZhbWlseTogIlNwYWNlIEdyb3Rlc2siLCAiQXZlbmlyIE5leHQiLCAiU2Vnb2UgVUkiLCAiUGluZ0ZhbmcgU0MiLCAiTm90byBTYW5zIENKSyBTQyIsIHNhbnMtc2VyaWY7CiAgICAgICAgZm9udC1zaXplOiAyNHB4OwogICAgICAgIGZvbnQtd2VpZ2h0OiA2MDA7CiAgICAgIH0KICAgICAgLmxhYmVsIHsgZmlsbDogIzE1MzI0ZDsgZm9udC1zaXplOiAyMnB4OyBmb250LXdlaWdodDogNjAwOyB9CiAgICAgIC5tdXRlZCB7IGZpbGw6ICM1YTc0OGY7IGZvbnQtc2l6ZTogMThweDsgfQogICAgICAucGFuZWwgeyBmaWxsOiAjZmZmZmZmOyBzdHJva2U6ICNkNWRlZTg7IHN0cm9rZS13aWR0aDogMS41OyB9CiAgICAgIC5ub2RlIHsgZmlsbDogI2ZmZmZmZjsgc3Ryb2tlOiAjNWE3NDhmOyBzdHJva2Utd2lkdGg6IDEuNTsgfQogICAgICAuY29ubmVjdG9yIHsgZmlsbDogbm9uZTsgc3Ryb2tlOiAjMzljNmY0OyBzdHJva2Utb3BhY2l0eTogMC43NTsgc3Ryb2tlLXdpZHRoOiAyLjU7IG1hcmtlci1lbmQ6IHVybCgjYXJyb3cpOyB9CiAgICAgIC5yZXR1cm4geyBmaWxsOiBub25lOyBzdHJva2U6ICMxMGQwNWQ7IHN0cm9rZS13aWR0aDogMi41OyBzdHJva2UtZGFzaGFycmF5OiA3IDU7IG1hcmtlci1lbmQ6IHVybCgjcmV0dXJuLWFycm93KTsgfQogICAgICAuZ3JlZW4geyBmaWxsOiAjMGE4ZjNkOyBmb250LXNpemU6IDE4cHg7IH0KICAgIDwvc3R5bGU+CiAgPC9kZWZzPgogIDxyZWN0IHdpZHRoPSIxMjAwIiBoZWlnaHQ9IjcyMCIgcng9IjE2IiBmaWxsPSIjZjVmOGZiIi8+CiAgPCEtLSBMb2NhbCBleGVjdXRpb24gb24gYSBHUFUgbm9kZSAtLT4KICA8cmVjdCBjbGFzcz0icGFuZWwiIHg9IjI0IiB5PSIyNCIgd2lkdGg9IjExNTIiIGhlaWdodD0iMjMwIiByeD0iMTQiLz4KICA8dGV4dCBjbGFzcz0iaGVhZGluZyIgeD0iNDgiIHk9IjY0Ij7mnKzlnLAgR1BVPC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJtdXRlZCIgeD0iNDgiIHk9Ijk0Ij5HUFUg6IqC54K5PC90ZXh0PgoKICA8cmVjdCBjbGFzcz0ibm9kZSIgeD0iNjQiIHk9IjEyNiIgd2lkdGg9IjI4OCIgaGVpZ2h0PSI5MCIgcng9IjEwIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSIyMDgiIHk9IjE2NCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+5bqU55So6L+b56iLPC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJtdXRlZCIgeD0iMjA4IiB5PSIxOTQiIHRleHQtYW5jaG9yPSJtaWRkbGUiPkNVREEgLyB2TExNPC90ZXh0PgogIDxwYXRoIGNsYXNzPSJjb25uZWN0b3IiIGQ9Ik0gMzUyIDE3MSBIIDUyMCIvPgogIDx0ZXh0IGNsYXNzPSJtdXRlZCIgeD0iNDM2IiB5PSIxNTEiIHRleHQtYW5jaG9yPSJtaWRkbGUiPkNVREEg6LCD55SoPC90ZXh0PgoKICA8cmVjdCBjbGFzcz0ibm9kZSIgeD0iNTIwIiB5PSIxMjYiIHdpZHRoPSIzMDAiIGhlaWdodD0iOTAiIHJ4PSIxMCIvPgogIDx0ZXh0IGNsYXNzPSJsYWJlbCIgeD0iNjcwIiB5PSIxNzkiIHRleHQtYW5jaG9yPSJtaWRkbGUiPk5WSURJQSDpqbHliqg8L3RleHQ+CiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSA4MjAgMTcxIEggMTAwMCIvPgogIDxyZWN0IGNsYXNzPSJub2RlIiB4PSIxMDAwIiB5PSIxMjYiIHdpZHRoPSIxMzYiIGhlaWdodD0iOTAiIHJ4PSIxMCIvPgogIDx0ZXh0IGNsYXNzPSJsYWJlbCIgeD0iMTA2OCIgeT0iMTc5IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5HUFU8L3RleHQ+CgogIDwhLS0gUmVtb3RlIGV4ZWN1dGlvbiBhY3Jvc3MgdGhlIENsaWVudCBhbmQgU2VydmVyIG5vZGVzIC0tPgogIDxyZWN0IGNsYXNzPSJwYW5lbCIgeD0iMjQiIHk9IjI4NiIgd2lkdGg9IjExNTIiIGhlaWdodD0iNDEwIiByeD0iMTQiLz4KICA8dGV4dCBjbGFzcz0iaGVhZGluZyIgeD0iNDgiIHk9IjMyNiI+UmVtb3RlIEdQVTwvdGV4dD4KCiAgPHJlY3QgY2xhc3M9InBhbmVsIiB4PSI0NCIgeT0iMzQ0IiB3aWR0aD0iMzI4IiBoZWlnaHQ9IjMyOCIgcng9IjEyIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSI2NCIgeT0iMzg4Ij5DUFUgQ2xpZW50PC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJtdXRlZCIgeD0iNjQiIHk9IjQxOCI+5pegIEdQVSAvIOaXoCBOVklESUEg6amx5YqoPC90ZXh0PgogIDxyZWN0IGNsYXNzPSJub2RlIiB4PSI2NCIgeT0iNDM4IiB3aWR0aD0iMjg4IiBoZWlnaHQ9IjcwIiByeD0iMTAiLz4KICA8dGV4dCBjbGFzcz0ibGFiZWwiIHg9IjIwOCIgeT0iNDY4IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj7lupTnlKjov5vnqIs8L3RleHQ+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSIyMDgiIHk9IjQ5NCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+Q1VEQSAvIHZMTE08L3RleHQ+CiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSAyMDggNTA4IFYgNTcwIi8+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSIyMjYiIHk9IjU0OCI+Q1VEQSDosIPnlKg8L3RleHQ+CiAgPHJlY3QgY2xhc3M9Im5vZGUiIHg9IjY0IiB5PSI1NzAiIHdpZHRoPSIyODgiIGhlaWdodD0iNzgiIHJ4PSIxMCIvPgogIDx0ZXh0IGNsYXNzPSJsYWJlbCIgeD0iMjA4IiB5PSI2MTYiIHRleHQtYW5jaG9yPSJtaWRkbGUiPkx1cGluZSDlrqLmiLfnq6/lupM8L3RleHQ+CgogIDxyZWN0IGNsYXNzPSJwYW5lbCIgeD0iNjQwIiB5PSIzNDQiIHdpZHRoPSI1MzYiIGhlaWdodD0iMzI4IiByeD0iMTIiLz4KICA8dGV4dCBjbGFzcz0ibGFiZWwiIHg9IjY2MCIgeT0iMzg4Ij5HUFUgU2VydmVyPC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJtdXRlZCIgeD0iNjYwIiB5PSI0MTgiPuWuieijhSBOVklESUEg6amx5YqoPC90ZXh0PgogIDxyZWN0IGNsYXNzPSJub2RlIiB4PSI2NjAiIHk9IjU3MCIgd2lkdGg9IjE3MCIgaGVpZ2h0PSI3OCIgcng9IjEwIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSI3NDUiIHk9IjYwNCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+THVwaW5lPC90ZXh0PgogIDx0ZXh0IHg9Ijc0NSIgeT0iNjMyIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj7mnI3liqHnq688L3RleHQ+CiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSA4MzAgNjA5IEggODY2Ii8+CiAgPHJlY3QgY2xhc3M9Im5vZGUiIHg9Ijg2NiIgeT0iNTcwIiB3aWR0aD0iMTcwIiBoZWlnaHQ9Ijc4IiByeD0iMTAiLz4KICA8dGV4dCBjbGFzcz0ibGFiZWwiIHg9Ijk1MSIgeT0iNjA0IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5OVklESUE8L3RleHQ+CiAgPHRleHQgeD0iOTUxIiB5PSI2MzIiIHRleHQtYW5jaG9yPSJtaWRkbGUiPumpseWKqDwvdGV4dD4KICA8cGF0aCBjbGFzcz0iY29ubmVjdG9yIiBkPSJNIDEwMzYgNjA5IEggMTA3MiIvPgogIDxyZWN0IGNsYXNzPSJub2RlIiB4PSIxMDcyIiB5PSI1NzAiIHdpZHRoPSI4NCIgaGVpZ2h0PSI3OCIgcng9IjEwIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSIxMTE0IiB5PSI2MTciIHRleHQtYW5jaG9yPSJtaWRkbGUiPkdQVTwvdGV4dD4KCiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSAzNTIgNTkwIEggNjYwIi8+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSI1MDYiIHk9IjU1MCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+572R57uc6L2s5Y+RIENVREEg6LCD55SoPC90ZXh0PgogIDxwYXRoIGNsYXNzPSJyZXR1cm4iIGQ9Ik0gNjYwIDYzMCBIIDM1MiIvPgogIDx0ZXh0IGNsYXNzPSJncmVlbiIgeD0iNTA2IiB5PSI2NzkiIHRleHQtYW5jaG9yPSJtaWRkbGUiPuiwg+eUqOe7k+aenDwvdGV4dD4KPC9zdmc+Cg==#light-only" width="1200" height="720" class="img_ev3q"> <img decoding="async" loading="lazy" alt="本地 GPU 与 Remote GPU 的工作原理对比" src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIxMjAwIiBoZWlnaHQ9IjcyMCIgdmlld0JveD0iMCAwIDEyMDAgNzIwIiByb2xlPSJpbWciIGFyaWEtbGFiZWxsZWRieT0idGl0bGUgZGVzY3JpcHRpb24iPgogIDx0aXRsZSBpZD0idGl0bGUiPuacrOWcsCBHUFUg5LiOIFJlbW90ZSBHUFUg55qE5bel5L2c5Y6f55CG5a+55q+UPC90aXRsZT4KICA8ZGVzYyBpZD0iZGVzY3JpcHRpb24iPuacrOWcsOW6lOeUqOmAmui/hyBOVklESUEg6amx5Yqo6K6/6ZeuIEdQVeOAgui/nOeoi+W6lOeUqOWcqCBDUFUgQ2xpZW50IOi/kOihjO+8jOe7jyBMdXBpbmUg5a6i5oi356uv5bqT6L2s5Y+RIENVREEg6LCD55So77yM5ZyoIEdQVSBTZXJ2ZXIg5LiK5a6M5oiQ6K6h566X5bm26L+U5Zue57uT5p6c44CCPC9kZXNjPgogIDxkZWZzPgogICAgPG1hcmtlciBpZD0iYXJyb3ciIG1hcmtlcldpZHRoPSI4IiBtYXJrZXJIZWlnaHQ9IjgiIHJlZlg9IjciIHJlZlk9IjQiIG9yaWVudD0iYXV0byI+CiAgICAgIDxwYXRoIGQ9Ik0gMCAwIEwgOCA0IEwgMCA4IFoiIGZpbGw9IiMzOWM2ZjQiIGZpbGwtb3BhY2l0eT0iMC43NSIvPgogICAgPC9tYXJrZXI+CiAgICA8bWFya2VyIGlkPSJyZXR1cm4tYXJyb3ciIG1hcmtlcldpZHRoPSI4IiBtYXJrZXJIZWlnaHQ9IjgiIHJlZlg9IjciIHJlZlk9IjQiIG9yaWVudD0iYXV0byI+CiAgICAgIDxwYXRoIGQ9Ik0gMCAwIEwgOCA0IEwgMCA4IFoiIGZpbGw9IiMxMGQwNWQiLz4KICAgIDwvbWFya2VyPgogICAgPHN0eWxlPgogICAgICB0ZXh0IHsKICAgICAgICBmaWxsOiAjZDhlZWZiOwogICAgICAgIGZvbnQtZmFtaWx5OiAiU291cmNlIFNhbnMgMyIsICJOb3RvIFNhbnMiLCAiU2Vnb2UgVUkiLCAiUGluZ0ZhbmcgU0MiLCAiTm90byBTYW5zIENKSyBTQyIsICJNaWNyb3NvZnQgWWFIZWkiLCBzYW5zLXNlcmlmOwogICAgICAgIGZvbnQtc2l6ZTogMjBweDsKICAgICAgfQogICAgICAuaGVhZGluZyB7CiAgICAgICAgZmlsbDogI2YwZjZmZDsKICAgICAgICBmb250LWZhbWlseTogIlNwYWNlIEdyb3Rlc2siLCAiQXZlbmlyIE5leHQiLCAiU2Vnb2UgVUkiLCAiUGluZ0ZhbmcgU0MiLCAiTm90byBTYW5zIENKSyBTQyIsIHNhbnMtc2VyaWY7CiAgICAgICAgZm9udC1zaXplOiAyNHB4OwogICAgICAgIGZvbnQtd2VpZ2h0OiA2MDA7CiAgICAgIH0KICAgICAgLmxhYmVsIHsgZmlsbDogI2YwZjZmZDsgZm9udC1zaXplOiAyMnB4OyBmb250LXdlaWdodDogNjAwOyB9CiAgICAgIC5tdXRlZCB7IGZpbGw6ICM5ZWI2Y2U7IGZvbnQtc2l6ZTogMThweDsgfQogICAgICAucGFuZWwgeyBmaWxsOiAjMTcxZTI3OyBzdHJva2U6ICMyYTM0NDI7IHN0cm9rZS13aWR0aDogMS41OyB9CiAgICAgIC5ub2RlIHsgZmlsbDogIzE3MWUyNzsgc3Ryb2tlOiAjOWViNmNlOyBzdHJva2Utd2lkdGg6IDEuNTsgfQogICAgICAuY29ubmVjdG9yIHsgZmlsbDogbm9uZTsgc3Ryb2tlOiAjMzljNmY0OyBzdHJva2Utb3BhY2l0eTogMC43NTsgc3Ryb2tlLXdpZHRoOiAyLjU7IG1hcmtlci1lbmQ6IHVybCgjYXJyb3cpOyB9CiAgICAgIC5yZXR1cm4geyBmaWxsOiBub25lOyBzdHJva2U6ICMxMGQwNWQ7IHN0cm9rZS13aWR0aDogMi41OyBzdHJva2UtZGFzaGFycmF5OiA3IDU7IG1hcmtlci1lbmQ6IHVybCgjcmV0dXJuLWFycm93KTsgfQogICAgICAuZ3JlZW4geyBmaWxsOiAjMzVlMzdiOyBmb250LXNpemU6IDE4cHg7IH0KICAgIDwvc3R5bGU+CiAgPC9kZWZzPgogIDxyZWN0IHdpZHRoPSIxMjAwIiBoZWlnaHQ9IjcyMCIgcng9IjE2IiBmaWxsPSIjMGIwZjE0Ii8+CiAgPCEtLSBMb2NhbCBleGVjdXRpb24gb24gYSBHUFUgbm9kZSAtLT4KICA8cmVjdCBjbGFzcz0icGFuZWwiIHg9IjI0IiB5PSIyNCIgd2lkdGg9IjExNTIiIGhlaWdodD0iMjMwIiByeD0iMTQiLz4KICA8dGV4dCBjbGFzcz0iaGVhZGluZyIgeD0iNDgiIHk9IjY0Ij7mnKzlnLAgR1BVPC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJtdXRlZCIgeD0iNDgiIHk9Ijk0Ij5HUFUg6IqC54K5PC90ZXh0PgoKICA8cmVjdCBjbGFzcz0ibm9kZSIgeD0iNjQiIHk9IjEyNiIgd2lkdGg9IjI4OCIgaGVpZ2h0PSI5MCIgcng9IjEwIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSIyMDgiIHk9IjE2NCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+5bqU55So6L+b56iLPC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJtdXRlZCIgeD0iMjA4IiB5PSIxOTQiIHRleHQtYW5jaG9yPSJtaWRkbGUiPkNVREEgLyB2TExNPC90ZXh0PgogIDxwYXRoIGNsYXNzPSJjb25uZWN0b3IiIGQ9Ik0gMzUyIDE3MSBIIDUyMCIvPgogIDx0ZXh0IGNsYXNzPSJtdXRlZCIgeD0iNDM2IiB5PSIxNTEiIHRleHQtYW5jaG9yPSJtaWRkbGUiPkNVREEg6LCD55SoPC90ZXh0PgoKICA8cmVjdCBjbGFzcz0ibm9kZSIgeD0iNTIwIiB5PSIxMjYiIHdpZHRoPSIzMDAiIGhlaWdodD0iOTAiIHJ4PSIxMCIvPgogIDx0ZXh0IGNsYXNzPSJsYWJlbCIgeD0iNjcwIiB5PSIxNzkiIHRleHQtYW5jaG9yPSJtaWRkbGUiPk5WSURJQSDpqbHliqg8L3RleHQ+CiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSA4MjAgMTcxIEggMTAwMCIvPgogIDxyZWN0IGNsYXNzPSJub2RlIiB4PSIxMDAwIiB5PSIxMjYiIHdpZHRoPSIxMzYiIGhlaWdodD0iOTAiIHJ4PSIxMCIvPgogIDx0ZXh0IGNsYXNzPSJsYWJlbCIgeD0iMTA2OCIgeT0iMTc5IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5HUFU8L3RleHQ+CgogIDwhLS0gUmVtb3RlIGV4ZWN1dGlvbiBhY3Jvc3MgdGhlIENsaWVudCBhbmQgU2VydmVyIG5vZGVzIC0tPgogIDxyZWN0IGNsYXNzPSJwYW5lbCIgeD0iMjQiIHk9IjI4NiIgd2lkdGg9IjExNTIiIGhlaWdodD0iNDEwIiByeD0iMTQiLz4KICA8dGV4dCBjbGFzcz0iaGVhZGluZyIgeD0iNDgiIHk9IjMyNiI+UmVtb3RlIEdQVTwvdGV4dD4KCiAgPHJlY3QgY2xhc3M9InBhbmVsIiB4PSI0NCIgeT0iMzQ0IiB3aWR0aD0iMzI4IiBoZWlnaHQ9IjMyOCIgcng9IjEyIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSI2NCIgeT0iMzg4Ij5DUFUgQ2xpZW50PC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJtdXRlZCIgeD0iNjQiIHk9IjQxOCI+5pegIEdQVSAvIOaXoCBOVklESUEg6amx5YqoPC90ZXh0PgogIDxyZWN0IGNsYXNzPSJub2RlIiB4PSI2NCIgeT0iNDM4IiB3aWR0aD0iMjg4IiBoZWlnaHQ9IjcwIiByeD0iMTAiLz4KICA8dGV4dCBjbGFzcz0ibGFiZWwiIHg9IjIwOCIgeT0iNDY4IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj7lupTnlKjov5vnqIs8L3RleHQ+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSIyMDgiIHk9IjQ5NCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+Q1VEQSAvIHZMTE08L3RleHQ+CiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSAyMDggNTA4IFYgNTcwIi8+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSIyMjYiIHk9IjU0OCI+Q1VEQSDosIPnlKg8L3RleHQ+CiAgPHJlY3QgY2xhc3M9Im5vZGUiIHg9IjY0IiB5PSI1NzAiIHdpZHRoPSIyODgiIGhlaWdodD0iNzgiIHJ4PSIxMCIvPgogIDx0ZXh0IGNsYXNzPSJsYWJlbCIgeD0iMjA4IiB5PSI2MTYiIHRleHQtYW5jaG9yPSJtaWRkbGUiPkx1cGluZSDlrqLmiLfnq6/lupM8L3RleHQ+CgogIDxyZWN0IGNsYXNzPSJwYW5lbCIgeD0iNjQwIiB5PSIzNDQiIHdpZHRoPSI1MzYiIGhlaWdodD0iMzI4IiByeD0iMTIiLz4KICA8dGV4dCBjbGFzcz0ibGFiZWwiIHg9IjY2MCIgeT0iMzg4Ij5HUFUgU2VydmVyPC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJtdXRlZCIgeD0iNjYwIiB5PSI0MTgiPuWuieijhSBOVklESUEg6amx5YqoPC90ZXh0PgogIDxyZWN0IGNsYXNzPSJub2RlIiB4PSI2NjAiIHk9IjU3MCIgd2lkdGg9IjE3MCIgaGVpZ2h0PSI3OCIgcng9IjEwIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSI3NDUiIHk9IjYwNCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+THVwaW5lPC90ZXh0PgogIDx0ZXh0IHg9Ijc0NSIgeT0iNjMyIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj7mnI3liqHnq688L3RleHQ+CiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSA4MzAgNjA5IEggODY2Ii8+CiAgPHJlY3QgY2xhc3M9Im5vZGUiIHg9Ijg2NiIgeT0iNTcwIiB3aWR0aD0iMTcwIiBoZWlnaHQ9Ijc4IiByeD0iMTAiLz4KICA8dGV4dCBjbGFzcz0ibGFiZWwiIHg9Ijk1MSIgeT0iNjA0IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5OVklESUE8L3RleHQ+CiAgPHRleHQgeD0iOTUxIiB5PSI2MzIiIHRleHQtYW5jaG9yPSJtaWRkbGUiPumpseWKqDwvdGV4dD4KICA8cGF0aCBjbGFzcz0iY29ubmVjdG9yIiBkPSJNIDEwMzYgNjA5IEggMTA3MiIvPgogIDxyZWN0IGNsYXNzPSJub2RlIiB4PSIxMDcyIiB5PSI1NzAiIHdpZHRoPSI4NCIgaGVpZ2h0PSI3OCIgcng9IjEwIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSIxMTE0IiB5PSI2MTciIHRleHQtYW5jaG9yPSJtaWRkbGUiPkdQVTwvdGV4dD4KCiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSAzNTIgNTkwIEggNjYwIi8+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSI1MDYiIHk9IjU1MCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+572R57uc6L2s5Y+RIENVREEg6LCD55SoPC90ZXh0PgogIDxwYXRoIGNsYXNzPSJyZXR1cm4iIGQ9Ik0gNjYwIDYzMCBIIDM1MiIvPgogIDx0ZXh0IGNsYXNzPSJncmVlbiIgeD0iNTA2IiB5PSI2NzkiIHRleHQtYW5jaG9yPSJtaWRkbGUiPuiwg+eUqOe7k+aenDwvdGV4dD4KPC9zdmc+Cg==#dark-only" width="1200" height="720" class="img_ev3q"></p>
<p>应用进程运行在 CPU Client，客户端库将 CUDA 调用通过网络转发给 GPU Server，由服务端驱动访问 GPU 执行计算，再返回调用结果。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="2-hami-如何集成-remote-gpu">2. HAMi 如何集成 Remote GPU<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#2-hami-%E5%A6%82%E4%BD%95%E9%9B%86%E6%88%90-remote-gpu" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>在 HAMi 的 Remote GPU 中，Client 与 Server 分别部署在同一个集群的不同节点上，或者说可以指定集群中的 GPU 节点作为 Server 提供计算资源，应用程序可以运行在 CPU-Only 节点，通过网络远程访问 Server 上的 GPU。</p>
<p>将这套能力接入 Kubernetes，需要解决几个问题。哪些节点可以提供远程 GPU？一个 Pod 应该使用哪台 Server？应用启动时，客户端库和连接地址如何准备？HAMi 将这些工作接入了资源发现、调度和 Pod 注入流程。</p>
<p><img decoding="async" loading="lazy" alt="HAMi Remote GPU 控制面与远程 CUDA 调用架构" src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIxMjAwIiBoZWlnaHQ9IjgwMCIgdmlld0JveD0iMCAwIDEyMDAgODAwIiByb2xlPSJpbWciIGFyaWEtbGFiZWxsZWRieT0idGl0bGUgZGVzY3JpcHRpb24iPgogIDx0aXRsZSBpZD0idGl0bGUiPkhBTWkgUmVtb3RlIEdQVSDmjqfliLbpnaLkuI7ov5znqIsgQ1VEQSDosIPnlKjmnrbmnoQ8L3RpdGxlPgogIDxkZXNjIGlkPSJkZXNjcmlwdGlvbiI+SEFNaSDlj5HnjrAgR1BVIFNlcnZlciDlubbkuLogUG9kIOmAieaLqSBDbGllbnQg5ZKMIFNlcnZlcuOAgmluaXQg5a655Zmo5o+Q5L6b5a6i5oi356uv5bqT77yMQ1BVIENsaWVudCDkuK3nmoTlt6XkvZzotJ/ovb3pgJrov4fnvZHnu5zorr/pl64gTHVwaW5lIOacjeWKoeerr++8jOeUsSBHUFUgU2VydmVyIOeahOmpseWKqOS4jiBHUFUg5omn6KGM6K6h566X44CCPC9kZXNjPgogIDxkZWZzPgogICAgPG1hcmtlciBpZD0iYXJyb3ciIG1hcmtlcldpZHRoPSI4IiBtYXJrZXJIZWlnaHQ9IjgiIHJlZlg9IjciIHJlZlk9IjQiIG9yaWVudD0iYXV0byI+CiAgICAgIDxwYXRoIGQ9Ik0gMCAwIEwgOCA0IEwgMCA4IFoiIGZpbGw9IiMzOWM2ZjQiIGZpbGwtb3BhY2l0eT0iMC43NSIvPgogICAgPC9tYXJrZXI+CiAgICA8c3R5bGU+CiAgICAgIHRleHQgewogICAgICAgIGZpbGw6ICMxYTM5NTY7CiAgICAgICAgZm9udC1mYW1pbHk6ICJTb3VyY2UgU2FucyAzIiwgIk5vdG8gU2FucyIsICJTZWdvZSBVSSIsICJQaW5nRmFuZyBTQyIsICJOb3RvIFNhbnMgQ0pLIFNDIiwgIk1pY3Jvc29mdCBZYUhlaSIsIHNhbnMtc2VyaWY7CiAgICAgICAgZm9udC1zaXplOiAyMHB4OwogICAgICB9CiAgICAgIC5oZWFkaW5nIHsKICAgICAgICBmaWxsOiAjMTUzMjRkOwogICAgICAgIGZvbnQtZmFtaWx5OiAiU3BhY2UgR3JvdGVzayIsICJBdmVuaXIgTmV4dCIsICJTZWdvZSBVSSIsICJQaW5nRmFuZyBTQyIsICJOb3RvIFNhbnMgQ0pLIFNDIiwgc2Fucy1zZXJpZjsKICAgICAgICBmb250LXNpemU6IDI0cHg7CiAgICAgICAgZm9udC13ZWlnaHQ6IDYwMDsKICAgICAgfQogICAgICAubGFiZWwgeyBmaWxsOiAjMTUzMjRkOyBmb250LXNpemU6IDIycHg7IGZvbnQtd2VpZ2h0OiA2MDA7IH0KICAgICAgLm11dGVkIHsgZmlsbDogIzVhNzQ4ZjsgZm9udC1zaXplOiAxOHB4OyB9CiAgICAgIC5tb25vIHsgZm9udC1mYW1pbHk6ICJJQk0gUGxleCBNb25vIiwgIlNGTW9uby1SZWd1bGFyIiwgQ29uc29sYXMsIG1vbm9zcGFjZTsgZm9udC1zaXplOiAxOHB4OyB9CiAgICAgIC5wYW5lbCB7IGZpbGw6ICNmZmZmZmY7IHN0cm9rZTogI2Q1ZGVlODsgc3Ryb2tlLXdpZHRoOiAxLjU7IH0KICAgICAgLm5vZGUgeyBmaWxsOiAjZmZmZmZmOyBzdHJva2U6ICM1YTc0OGY7IHN0cm9rZS13aWR0aDogMS41OyB9CiAgICAgIC5oYW1pIHsgZmlsbDogIzEwZDA1ZDsgZmlsbC1vcGFjaXR5OiAwLjA4OyBzdHJva2U6ICMxMGQwNWQ7IHN0cm9rZS13aWR0aDogMjsgfQogICAgICAuc2NoZWR1bGVyIHsgZmlsbDogIzdjOGNmZjsgZmlsbC1vcGFjaXR5OiAwLjA4OyBzdHJva2U6ICM3YzhjZmY7IHN0cm9rZS13aWR0aDogMjsgfQogICAgICAuY29ubmVjdG9yIHsgZmlsbDogbm9uZTsgc3Ryb2tlOiAjMzljNmY0OyBzdHJva2Utb3BhY2l0eTogMC43NTsgc3Ryb2tlLXdpZHRoOiAyLjU7IG1hcmtlci1lbmQ6IHVybCgjYXJyb3cpOyB9CiAgICA8L3N0eWxlPgogIDwvZGVmcz4KICA8cmVjdCB3aWR0aD0iMTIwMCIgaGVpZ2h0PSI4MDAiIHJ4PSIxNiIgZmlsbD0iI2Y1ZjhmYiIvPgogIDwhLS0gVGhlIGNvbnRyb2wgcGxhbmUgc2VsZWN0cyB0aGUgU2VydmVyIGFuZCBwcmVwYXJlcyB0aGUgUG9kIC0tPgogIDxyZWN0IGNsYXNzPSJwYW5lbCIgeD0iMjQiIHk9IjI0IiB3aWR0aD0iMTE1MiIgaGVpZ2h0PSIyMTAiIHJ4PSIxNCIvPgogIDx0ZXh0IGNsYXNzPSJoZWFkaW5nIiB4PSI0OCIgeT0iNjQiPuaOp+WItumdojwvdGV4dD4KICA8dGV4dCBjbGFzcz0ibXV0ZWQiIHg9IjQ4IiB5PSI5NiI+6YCJ5oupIEdQVSBTZXJ2ZXI8L3RleHQ+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSI0OCIgeT0iMTI0Ij7kuI4gQ2xpZW50IOiKgueCuTwvdGV4dD4KICA8cmVjdCBjbGFzcz0ic2NoZWR1bGVyIiB4PSIzOTAiIHk9Ijg0IiB3aWR0aD0iNDIwIiBoZWlnaHQ9IjEwMCIgcng9IjEwIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSI2MDAiIHk9IjEyNiIgdGV4dC1hbmNob3I9Im1pZGRsZSI+SEFNaSDosIPluqblmag8L3RleHQ+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSI2MDAiIHk9IjE1OCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+cmVtb3RlLWdwdSDlkI7nq688L3RleHQ+CgogIDx0ZXh0IGNsYXNzPSJoZWFkaW5nIiB4PSIyNCIgeT0iMjgyIj7mlbDmja7pnaI8L3RleHQ+CgogIDwhLS0gQ1BVLW9ubHkgQ2xpZW50IGFuZCBpdHMgaW5qZWN0ZWQgbGlicmFyaWVzIC0tPgogIDxyZWN0IGNsYXNzPSJwYW5lbCIgeD0iMjQiIHk9IjMxMiIgd2lkdGg9IjUwMCIgaGVpZ2h0PSI0NjAiIHJ4PSIxNCIvPgogIDx0ZXh0IGNsYXNzPSJoZWFkaW5nIiB4PSI0OCIgeT0iMzUyIj5DbGllbnQg6IqC54K5PC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJtdXRlZCIgeD0iNDgiIHk9IjM4NCI+5pegIEdQVSAvIOaXoCBOVklESUEg6amx5YqoPC90ZXh0PgogIDxyZWN0IGNsYXNzPSJub2RlIiB4PSI0OCIgeT0iNDEyIiB3aWR0aD0iNDUyIiBoZWlnaHQ9IjMyOCIgcng9IjEyIiBzdHJva2UtZGFzaGFycmF5PSI5IDYiLz4KICA8dGV4dCBjbGFzcz0ibGFiZWwiIHg9IjcwIiB5PSI0NDgiPlBvZDwvdGV4dD4KICA8dGV4dCBjbGFzcz0ibW9ubyIgeD0iNzAiIHk9IjQ3OCI+bnZpZGlhLmNvbS9yZW1vdGUtZ3B1OiAxPC90ZXh0PgoKICA8cmVjdCBjbGFzcz0iaGFtaSIgeD0iNzYiIHk9IjUxOCIgd2lkdGg9IjM5NiIgaGVpZ2h0PSI3NiIgcng9IjEwIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSIyNzQiIHk9IjU1MCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+aW5pdCDlrrnlmag8L3RleHQ+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSIyNzQiIHk9IjU3OCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+THVwaW5lIOWuouaIt+err+W6kyArIGxpYnZncHUuc288L3RleHQ+CiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSAxNTAgNTk0IFYgNjQ4Ii8+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSIxNzYiIHk9IjYyOCI+5o+Q5L6b5a6i5oi356uv5bqTPC90ZXh0PgogIDxyZWN0IGNsYXNzPSJub2RlIiB4PSI3NiIgeT0iNjQ4IiB3aWR0aD0iMzk2IiBoZWlnaHQ9IjY4IiByeD0iMTAiLz4KICA8dGV4dCBjbGFzcz0ibGFiZWwiIHg9IjI3NCIgeT0iNjc3IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj7lt6XkvZzotJ/ovb3lrrnlmag8L3RleHQ+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSIyNzQiIHk9IjcwMyIgdGV4dC1hbmNob3I9Im1pZGRsZSI+Q1VEQSAvIHZMTE08L3RleHQ+CgogIDwhLS0gR1BVIFNlcnZlciBkaXNjb3ZlcnkgYW5kIEdQVSByZWdpc3RyYXRpb24gLS0+CiAgPHJlY3QgY2xhc3M9InBhbmVsIiB4PSI2NjAiIHk9IjMxMiIgd2lkdGg9IjUxNiIgaGVpZ2h0PSI0NjAiIHJ4PSIxNCIvPgogIDx0ZXh0IGNsYXNzPSJoZWFkaW5nIiB4PSI2ODQiIHk9IjM1MiI+R1BVIFNlcnZlciDoioLngrk8L3RleHQ+CiAgPHRleHQgY2xhc3M9Im1vbm8iIHg9IjY4NCIgeT0iMzg0Ij5vcGVyYXRpbmdtb2RlOiByZW1vdGU8L3RleHQ+CiAgPHJlY3QgY2xhc3M9ImhhbWkiIHg9IjkwOCIgeT0iNDQ0IiB3aWR0aD0iMjQ0IiBoZWlnaHQ9IjEwMCIgcng9IjEwIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSIxMDMwIiB5PSI0ODMiIHRleHQtYW5jaG9yPSJtaWRkbGUiIHN0eWxlPSJmb250LXNpemU6IDIwcHgiPkhBTWkgRGV2aWNlIFBsdWdpbjwvdGV4dD4KICA8dGV4dCBjbGFzcz0ibXV0ZWQiIHg9IjEwMzAiIHk9IjUxNSIgdGV4dC1hbmNob3I9Im1pZGRsZSI+6IqC54K55qCH562+IC8gR1BVIOazqOWGjDwvdGV4dD4KCiAgPHJlY3QgY2xhc3M9Im5vZGUiIHg9IjY4NCIgeT0iNTcwIiB3aWR0aD0iMjM0IiBoZWlnaHQ9Ijg2IiByeD0iMTAiLz4KICA8dGV4dCBjbGFzcz0ibGFiZWwiIHg9IjgwMSIgeT0iNjA3IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5MdXBpbmUg5pyN5Yqh56uvPC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJtb25vIiB4PSI4MDEiIHk9IjYzNyIgdGV4dC1hbmNob3I9Im1pZGRsZSI+VENQIDE0ODMzPC90ZXh0PgogIDxyZWN0IGNsYXNzPSJub2RlIiB4PSI3NTYiIHk9IjY4NCIgd2lkdGg9IjMyOCIgaGVpZ2h0PSI2NCIgcng9IjEwIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSI5MjAiIHk9IjcyNCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+TlZJRElBIEdQVSDkuI7pqbHliqg8L3RleHQ+CiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSA4MDEgNjU2IFYgNjY4IEggODUwIFYgNjg0Ii8+CiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSAxMDMwIDU0NCBWIDY2OCBIIDEwMDAgViA2ODQiLz4KCiAgPCEtLSBUaGUgc2NoZWR1bGVyIGRpc2NvdmVycyBhIHBvb2wgYW5kIGJpbmRzIHRoZSB3b3JrbG9hZCAtLT4KICA8cGF0aCBjbGFzcz0iY29ubmVjdG9yIiBkPSJNIDc0MCAxODQgViAyNjYgSCAxMTAwIFYgMzEyIi8+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSI5MzAiIHk9IjI0NSIgdGV4dC1hbmNob3I9Im1pZGRsZSI+5Y+R546w5bim5qCH562+55qEIEx1cGluZSDotYTmupDmsaA8L3RleHQ+CiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSA0ODAgMTg0IFYgNDEyIi8+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSI0NTQiIHk9IjI1NSIgdGV4dC1hbmNob3I9ImVuZCI+57uR5a6aIFBvZO+8jOazqOWFpTwvdGV4dD4KICA8dGV4dCBjbGFzcz0ibW9ubyIgeD0iNDU0IiB5PSIyODQiIHRleHQtYW5jaG9yPSJlbmQiPkxVUElORV9TRVJWRVI8L3RleHQ+CgogIDwhLS0gQ1VEQSBjYWxscyB0cmF2ZWwgZnJvbSB0aGUgYXBwbGljYXRpb24gdG8gdGhlIHJlbW90ZSBlbmRwb2ludCAtLT4KICA8cGF0aCBjbGFzcz0iY29ubmVjdG9yIiBkPSJNIDQ3MiA2ODIgSCA2NDAgViA2MTMgSCA2ODQiLz4KICA8dGV4dCBjbGFzcz0ibXV0ZWQiIHg9IjU4MiIgeT0iNjQwIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj7nvZHnu5w8L3RleHQ+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSI1ODIiIHk9IjY2NSIgdGV4dC1hbmNob3I9Im1pZGRsZSI+Q1VEQSDosIPnlKg8L3RleHQ+Cjwvc3ZnPgo=#light-only" width="1200" height="800" class="img_ev3q"> <img decoding="async" loading="lazy" alt="HAMi Remote GPU 控制面与远程 CUDA 调用架构" src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIxMjAwIiBoZWlnaHQ9IjgwMCIgdmlld0JveD0iMCAwIDEyMDAgODAwIiByb2xlPSJpbWciIGFyaWEtbGFiZWxsZWRieT0idGl0bGUgZGVzY3JpcHRpb24iPgogIDx0aXRsZSBpZD0idGl0bGUiPkhBTWkgUmVtb3RlIEdQVSDmjqfliLbpnaLkuI7ov5znqIsgQ1VEQSDosIPnlKjmnrbmnoQ8L3RpdGxlPgogIDxkZXNjIGlkPSJkZXNjcmlwdGlvbiI+SEFNaSDlj5HnjrAgR1BVIFNlcnZlciDlubbkuLogUG9kIOmAieaLqSBDbGllbnQg5ZKMIFNlcnZlcuOAgmluaXQg5a655Zmo5o+Q5L6b5a6i5oi356uv5bqT77yMQ1BVIENsaWVudCDkuK3nmoTlt6XkvZzotJ/ovb3pgJrov4fnvZHnu5zorr/pl64gTHVwaW5lIOacjeWKoeerr++8jOeUsSBHUFUgU2VydmVyIOeahOmpseWKqOS4jiBHUFUg5omn6KGM6K6h566X44CCPC9kZXNjPgogIDxkZWZzPgogICAgPG1hcmtlciBpZD0iYXJyb3ciIG1hcmtlcldpZHRoPSI4IiBtYXJrZXJIZWlnaHQ9IjgiIHJlZlg9IjciIHJlZlk9IjQiIG9yaWVudD0iYXV0byI+CiAgICAgIDxwYXRoIGQ9Ik0gMCAwIEwgOCA0IEwgMCA4IFoiIGZpbGw9IiMzOWM2ZjQiIGZpbGwtb3BhY2l0eT0iMC43NSIvPgogICAgPC9tYXJrZXI+CiAgICA8c3R5bGU+CiAgICAgIHRleHQgewogICAgICAgIGZpbGw6ICNkOGVlZmI7CiAgICAgICAgZm9udC1mYW1pbHk6ICJTb3VyY2UgU2FucyAzIiwgIk5vdG8gU2FucyIsICJTZWdvZSBVSSIsICJQaW5nRmFuZyBTQyIsICJOb3RvIFNhbnMgQ0pLIFNDIiwgIk1pY3Jvc29mdCBZYUhlaSIsIHNhbnMtc2VyaWY7CiAgICAgICAgZm9udC1zaXplOiAyMHB4OwogICAgICB9CiAgICAgIC5oZWFkaW5nIHsKICAgICAgICBmaWxsOiAjZjBmNmZkOwogICAgICAgIGZvbnQtZmFtaWx5OiAiU3BhY2UgR3JvdGVzayIsICJBdmVuaXIgTmV4dCIsICJTZWdvZSBVSSIsICJQaW5nRmFuZyBTQyIsICJOb3RvIFNhbnMgQ0pLIFNDIiwgc2Fucy1zZXJpZjsKICAgICAgICBmb250LXNpemU6IDI0cHg7CiAgICAgICAgZm9udC13ZWlnaHQ6IDYwMDsKICAgICAgfQogICAgICAubGFiZWwgeyBmaWxsOiAjZjBmNmZkOyBmb250LXNpemU6IDIycHg7IGZvbnQtd2VpZ2h0OiA2MDA7IH0KICAgICAgLm11dGVkIHsgZmlsbDogIzllYjZjZTsgZm9udC1zaXplOiAxOHB4OyB9CiAgICAgIC5tb25vIHsgZm9udC1mYW1pbHk6ICJJQk0gUGxleCBNb25vIiwgIlNGTW9uby1SZWd1bGFyIiwgQ29uc29sYXMsIG1vbm9zcGFjZTsgZm9udC1zaXplOiAxOHB4OyB9CiAgICAgIC5wYW5lbCB7IGZpbGw6ICMxNzFlMjc7IHN0cm9rZTogIzJhMzQ0Mjsgc3Ryb2tlLXdpZHRoOiAxLjU7IH0KICAgICAgLm5vZGUgeyBmaWxsOiAjMTcxZTI3OyBzdHJva2U6ICM5ZWI2Y2U7IHN0cm9rZS13aWR0aDogMS41OyB9CiAgICAgIC5oYW1pIHsgZmlsbDogIzEwZDA1ZDsgZmlsbC1vcGFjaXR5OiAwLjA4OyBzdHJva2U6ICMxMGQwNWQ7IHN0cm9rZS13aWR0aDogMjsgfQogICAgICAuc2NoZWR1bGVyIHsgZmlsbDogIzdjOGNmZjsgZmlsbC1vcGFjaXR5OiAwLjA4OyBzdHJva2U6ICM3YzhjZmY7IHN0cm9rZS13aWR0aDogMjsgfQogICAgICAuY29ubmVjdG9yIHsgZmlsbDogbm9uZTsgc3Ryb2tlOiAjMzljNmY0OyBzdHJva2Utb3BhY2l0eTogMC43NTsgc3Ryb2tlLXdpZHRoOiAyLjU7IG1hcmtlci1lbmQ6IHVybCgjYXJyb3cpOyB9CiAgICA8L3N0eWxlPgogIDwvZGVmcz4KICA8cmVjdCB3aWR0aD0iMTIwMCIgaGVpZ2h0PSI4MDAiIHJ4PSIxNiIgZmlsbD0iIzBiMGYxNCIvPgogIDwhLS0gVGhlIGNvbnRyb2wgcGxhbmUgc2VsZWN0cyB0aGUgU2VydmVyIGFuZCBwcmVwYXJlcyB0aGUgUG9kIC0tPgogIDxyZWN0IGNsYXNzPSJwYW5lbCIgeD0iMjQiIHk9IjI0IiB3aWR0aD0iMTE1MiIgaGVpZ2h0PSIyMTAiIHJ4PSIxNCIvPgogIDx0ZXh0IGNsYXNzPSJoZWFkaW5nIiB4PSI0OCIgeT0iNjQiPuaOp+WItumdojwvdGV4dD4KICA8dGV4dCBjbGFzcz0ibXV0ZWQiIHg9IjQ4IiB5PSI5NiI+6YCJ5oupIEdQVSBTZXJ2ZXI8L3RleHQ+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSI0OCIgeT0iMTI0Ij7kuI4gQ2xpZW50IOiKgueCuTwvdGV4dD4KICA8cmVjdCBjbGFzcz0ic2NoZWR1bGVyIiB4PSIzOTAiIHk9Ijg0IiB3aWR0aD0iNDIwIiBoZWlnaHQ9IjEwMCIgcng9IjEwIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSI2MDAiIHk9IjEyNiIgdGV4dC1hbmNob3I9Im1pZGRsZSI+SEFNaSDosIPluqblmag8L3RleHQ+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSI2MDAiIHk9IjE1OCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+cmVtb3RlLWdwdSDlkI7nq688L3RleHQ+CgogIDx0ZXh0IGNsYXNzPSJoZWFkaW5nIiB4PSIyNCIgeT0iMjgyIj7mlbDmja7pnaI8L3RleHQ+CgogIDwhLS0gQ1BVLW9ubHkgQ2xpZW50IGFuZCBpdHMgaW5qZWN0ZWQgbGlicmFyaWVzIC0tPgogIDxyZWN0IGNsYXNzPSJwYW5lbCIgeD0iMjQiIHk9IjMxMiIgd2lkdGg9IjUwMCIgaGVpZ2h0PSI0NjAiIHJ4PSIxNCIvPgogIDx0ZXh0IGNsYXNzPSJoZWFkaW5nIiB4PSI0OCIgeT0iMzUyIj5DbGllbnQg6IqC54K5PC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJtdXRlZCIgeD0iNDgiIHk9IjM4NCI+5pegIEdQVSAvIOaXoCBOVklESUEg6amx5YqoPC90ZXh0PgogIDxyZWN0IGNsYXNzPSJub2RlIiB4PSI0OCIgeT0iNDEyIiB3aWR0aD0iNDUyIiBoZWlnaHQ9IjMyOCIgcng9IjEyIiBzdHJva2UtZGFzaGFycmF5PSI5IDYiLz4KICA8dGV4dCBjbGFzcz0ibGFiZWwiIHg9IjcwIiB5PSI0NDgiPlBvZDwvdGV4dD4KICA8dGV4dCBjbGFzcz0ibW9ubyIgeD0iNzAiIHk9IjQ3OCI+bnZpZGlhLmNvbS9yZW1vdGUtZ3B1OiAxPC90ZXh0PgoKICA8cmVjdCBjbGFzcz0iaGFtaSIgeD0iNzYiIHk9IjUxOCIgd2lkdGg9IjM5NiIgaGVpZ2h0PSI3NiIgcng9IjEwIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSIyNzQiIHk9IjU1MCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+aW5pdCDlrrnlmag8L3RleHQ+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSIyNzQiIHk9IjU3OCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+THVwaW5lIOWuouaIt+err+W6kyArIGxpYnZncHUuc288L3RleHQ+CiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSAxNTAgNTk0IFYgNjQ4Ii8+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSIxNzYiIHk9IjYyOCI+5o+Q5L6b5a6i5oi356uv5bqTPC90ZXh0PgogIDxyZWN0IGNsYXNzPSJub2RlIiB4PSI3NiIgeT0iNjQ4IiB3aWR0aD0iMzk2IiBoZWlnaHQ9IjY4IiByeD0iMTAiLz4KICA8dGV4dCBjbGFzcz0ibGFiZWwiIHg9IjI3NCIgeT0iNjc3IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj7lt6XkvZzotJ/ovb3lrrnlmag8L3RleHQ+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSIyNzQiIHk9IjcwMyIgdGV4dC1hbmNob3I9Im1pZGRsZSI+Q1VEQSAvIHZMTE08L3RleHQ+CgogIDwhLS0gR1BVIFNlcnZlciBkaXNjb3ZlcnkgYW5kIEdQVSByZWdpc3RyYXRpb24gLS0+CiAgPHJlY3QgY2xhc3M9InBhbmVsIiB4PSI2NjAiIHk9IjMxMiIgd2lkdGg9IjUxNiIgaGVpZ2h0PSI0NjAiIHJ4PSIxNCIvPgogIDx0ZXh0IGNsYXNzPSJoZWFkaW5nIiB4PSI2ODQiIHk9IjM1MiI+R1BVIFNlcnZlciDoioLngrk8L3RleHQ+CiAgPHRleHQgY2xhc3M9Im1vbm8iIHg9IjY4NCIgeT0iMzg0Ij5vcGVyYXRpbmdtb2RlOiByZW1vdGU8L3RleHQ+CiAgPHJlY3QgY2xhc3M9ImhhbWkiIHg9IjkwOCIgeT0iNDQ0IiB3aWR0aD0iMjQ0IiBoZWlnaHQ9IjEwMCIgcng9IjEwIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSIxMDMwIiB5PSI0ODMiIHRleHQtYW5jaG9yPSJtaWRkbGUiIHN0eWxlPSJmb250LXNpemU6IDIwcHgiPkhBTWkgRGV2aWNlIFBsdWdpbjwvdGV4dD4KICA8dGV4dCBjbGFzcz0ibXV0ZWQiIHg9IjEwMzAiIHk9IjUxNSIgdGV4dC1hbmNob3I9Im1pZGRsZSI+6IqC54K55qCH562+IC8gR1BVIOazqOWGjDwvdGV4dD4KCiAgPHJlY3QgY2xhc3M9Im5vZGUiIHg9IjY4NCIgeT0iNTcwIiB3aWR0aD0iMjM0IiBoZWlnaHQ9Ijg2IiByeD0iMTAiLz4KICA8dGV4dCBjbGFzcz0ibGFiZWwiIHg9IjgwMSIgeT0iNjA3IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5MdXBpbmUg5pyN5Yqh56uvPC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJtb25vIiB4PSI4MDEiIHk9IjYzNyIgdGV4dC1hbmNob3I9Im1pZGRsZSI+VENQIDE0ODMzPC90ZXh0PgogIDxyZWN0IGNsYXNzPSJub2RlIiB4PSI3NTYiIHk9IjY4NCIgd2lkdGg9IjMyOCIgaGVpZ2h0PSI2NCIgcng9IjEwIi8+CiAgPHRleHQgY2xhc3M9ImxhYmVsIiB4PSI5MjAiIHk9IjcyNCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+TlZJRElBIEdQVSDkuI7pqbHliqg8L3RleHQ+CiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSA4MDEgNjU2IFYgNjY4IEggODUwIFYgNjg0Ii8+CiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSAxMDMwIDU0NCBWIDY2OCBIIDEwMDAgViA2ODQiLz4KCiAgPCEtLSBUaGUgc2NoZWR1bGVyIGRpc2NvdmVycyBhIHBvb2wgYW5kIGJpbmRzIHRoZSB3b3JrbG9hZCAtLT4KICA8cGF0aCBjbGFzcz0iY29ubmVjdG9yIiBkPSJNIDc0MCAxODQgViAyNjYgSCAxMTAwIFYgMzEyIi8+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSI5MzAiIHk9IjI0NSIgdGV4dC1hbmNob3I9Im1pZGRsZSI+5Y+R546w5bim5qCH562+55qEIEx1cGluZSDotYTmupDmsaA8L3RleHQ+CiAgPHBhdGggY2xhc3M9ImNvbm5lY3RvciIgZD0iTSA0ODAgMTg0IFYgNDEyIi8+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSI0NTQiIHk9IjI1NSIgdGV4dC1hbmNob3I9ImVuZCI+57uR5a6aIFBvZO+8jOazqOWFpTwvdGV4dD4KICA8dGV4dCBjbGFzcz0ibW9ubyIgeD0iNDU0IiB5PSIyODQiIHRleHQtYW5jaG9yPSJlbmQiPkxVUElORV9TRVJWRVI8L3RleHQ+CgogIDwhLS0gQ1VEQSBjYWxscyB0cmF2ZWwgZnJvbSB0aGUgYXBwbGljYXRpb24gdG8gdGhlIHJlbW90ZSBlbmRwb2ludCAtLT4KICA8cGF0aCBjbGFzcz0iY29ubmVjdG9yIiBkPSJNIDQ3MiA2ODIgSCA2NDAgViA2MTMgSCA2ODQiLz4KICA8dGV4dCBjbGFzcz0ibXV0ZWQiIHg9IjU4MiIgeT0iNjQwIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj7nvZHnu5w8L3RleHQ+CiAgPHRleHQgY2xhc3M9Im11dGVkIiB4PSI1ODIiIHk9IjY2NSIgdGV4dC1hbmNob3I9Im1pZGRsZSI+Q1VEQSDosIPnlKg8L3RleHQ+Cjwvc3ZnPgo=#dark-only" width="1200" height="800" class="img_ev3q"></p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="21-将-gpu-节点注册为-server">2.1 将 GPU 节点注册为 Server<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#21-%E5%B0%86-gpu-%E8%8A%82%E7%82%B9%E6%B3%A8%E5%86%8C%E4%B8%BA-server" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>管理员通过 <code>operatingmode: remote</code> 指定 GPU 节点的运行模式。Device Plugin 将 GPU 信息注册为远程资源，并自动为节点添加 <code>hami.io/lupine-server</code> 发现标签，供 HAMi 选择 GPU Server。远程服务端在 GPU 节点运行，为后续工作负载提供 CUDA 调用入口。</p>
<p>这样，HAMi 就能发现集群中可供分配的 GPU Server。服务器还可以按节点标签划分资源池，供不同工作负载选择。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="22-为-pod-选择-client-和-gpu-server">2.2 为 Pod 选择 Client 和 GPU Server<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#22-%E4%B8%BA-pod-%E9%80%89%E6%8B%A9-client-%E5%92%8C-gpu-server" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>工作负载通过 <code>nvidia.com/remote-gpu</code> 声明远程 GPU 需求，还可以通过 <code>nvidia.com/remote-gpu-memory</code> 指定每张卡的显存配额，单位为 MiB。</p>
<p>下面摘录 Pod 的关键配置，将应用放到 CPU 节点，申请 1 张远程 GPU，并设置每张卡的显存配额为 3000 MiB。</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">spec</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">nodeSelector</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">kubernetes.io/hostname</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> cpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">client</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token number" style="color:rgb(247, 140, 108)">01</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">containers</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> cuda</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">image</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> nvidia/cuda</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">13.0.2</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">devel</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">ubuntu24.04</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">command</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">[</span><span class="token string" style="color:rgb(195, 232, 141)">"nvidia-smi"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"-L"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">]</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token key atrule">limits</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">nvidia.com/remote-gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 远程 GPU 数量</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">nvidia.com/remote-gpu-memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">3000</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 每张 GPU 的显存配额，单位 MiB</span><br></div></code></pre></div></div>
<p>调度时需要确定两个位置。一处是 Pod 实际运行的 Client 节点，另一处是提供 GPU 的 Server。本文通过 <code>nodeSelector</code> 将 Pod 放在指定 CPU 节点上，再由 HAMi 为它分配远端 Server。</p>
<blockquote>
<p>如果需要选择特定资源池，Pod 可以设置 <code>hami.io/remote-server</code> 注解。</p>
</blockquote>
<div class="theme-admonition theme-admonition-warning admonition_xJq3 alert alert--warning"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 16 16"><path fill-rule="evenodd" d="M8.893 1.5c-.183-.31-.52-.5-.887-.5s-.703.19-.886.5L.138 13.499a.98.98 0 0 0 0 1.001c.193.31.53.501.886.501h13.964c.367 0 .704-.19.877-.5a1.03 1.03 0 0 0 .01-1.002L8.893 1.5zm.133 11.497H6.987v-2.003h2.039v2.003zm0-3.004H6.987V5.987h2.039v4.006z"></path></svg></span>当前版本限制</div><div class="admonitionContent_BuS1"><p>一个工作负载独占整个 GPU 服务端。GPU 数量应设为该服务端的物理卡数，因为端点会暴露其全部 GPU。设置较小显存限额不会让其他工作负载共享剩余服务端容量。本文后续测试 Pod 应依次运行，或分别使用不同 GPU 服务端。</p></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="23-为应用准备客户端环境">2.3 为应用准备客户端环境<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#23-%E4%B8%BA%E5%BA%94%E7%94%A8%E5%87%86%E5%A4%87%E5%AE%A2%E6%88%B7%E7%AB%AF%E7%8E%AF%E5%A2%83" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>应用被分配到远程 GPU 后，还需要能加载客户端库，并知道连接哪台 Server。</p>
<p>HAMi 通过 init 容器下发 Lupine 客户端库和兼容的 <code>nvidia-smi</code>，再将库挂载到应用容器中，同时通过 <code>LUPINE_SERVER</code> 环境变量注入所选 Server 的连接地址。启用显存限制时，还会下发 HAMi-core，并通过 <code>LD_PRELOAD</code> 将其加入调用链，执行显存配额限制。</p>
<p>下面以启用显存配额的工作负载为例，摘录 HAMi 注入后的 init 容器以及关键信息：</p>
<blockquote>
<p>这些字段由 HAMi 注入，创建工作负载时不需要额外填写。</p>
</blockquote>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">spec</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">initContainers</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> hami</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">remote</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">lib</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">image</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> docker.m.daocloud.io/projecthami/hami</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">v2.11.0</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">volumeMounts</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> hami</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">remote</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">lib</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">mountPath</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> /hami</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">remote</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">gpu</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> hami</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">lupine</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">client</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">image</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> ghcr.io/lupinemachines/lupine</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">client</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">cuda</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">13.3.1</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">ubuntu24.04</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">volumeMounts</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> hami</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">lupine</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">client</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">mountPath</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> /lupine</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">lib</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">containers</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">env</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> LUPINE_SERVER</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">valueFrom</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">            </span><span class="token key atrule">fieldRef</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">              </span><span class="token key atrule">fieldPath</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> metadata.annotations</span><span class="token punctuation" style="color:rgb(199, 146, 234)">[</span><span class="token string" style="color:rgb(195, 232, 141)">'hami.io/lupine-endpoint'</span><span class="token punctuation" style="color:rgb(199, 146, 234)">]</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> LD_PRELOAD</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">value</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> /hami</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">remote</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">gpu/libvgpu.so</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> CUDA_DEVICE_MEMORY_LIMIT</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">value</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> 15360m</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">volumeMounts</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> hami</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">remote</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">lib</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">mountPath</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> /hami</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">remote</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">gpu</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> hami</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">lupine</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">client</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">mountPath</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> /opt/lupine/lib</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">readOnly</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token boolean important" style="color:rgb(255, 88, 116)">true</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">volumes</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> hami</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">remote</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">lib</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">emptyDir</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> hami</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">lupine</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">client</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">emptyDir</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><br></div></code></pre></div></div>
<table><thead><tr><th>init 容器</th><th>作用</th><th>退出码</th></tr></thead><tbody><tr><td><code>hami-remote-gpu-lib</code></td><td>将 HAMi-core 的 <code>libvgpu.so</code> 复制到共享卷</td><td>0</td></tr><tr><td><code>hami-lupine-client</code></td><td>将 Lupine 客户端库、依赖库和兼容的 <code>nvidia-smi</code> 复制到共享卷</td><td>0</td></tr></tbody></table>
<p>两个共享卷均为 <code>emptyDir</code>。init 容器先写入文件，应用容器随后挂载使用。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="24-启动应用并执行远程-cuda-调用">2.4 启动应用并执行远程 CUDA 调用<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#24-%E5%90%AF%E5%8A%A8%E5%BA%94%E7%94%A8%E5%B9%B6%E6%89%A7%E8%A1%8C%E8%BF%9C%E7%A8%8B-cuda-%E8%B0%83%E7%94%A8" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>客户端环境准备完成后，应用在 CPU Client 上启动。应用发起 CUDA 调用，经客户端库转发到已分配的 GPU Server，由服务端访问 GPU 完成计算，并返回调用结果。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="3-环境准备与部署">3. 环境准备与部署<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#3-%E7%8E%AF%E5%A2%83%E5%87%86%E5%A4%87%E4%B8%8E%E9%83%A8%E7%BD%B2" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="31-准备-kubernetes-集群">3.1 准备 Kubernetes 集群<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#31-%E5%87%86%E5%A4%87-kubernetes-%E9%9B%86%E7%BE%A4" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>准备一个包含 GPU 节点和纯 CPU 节点的 Kubernetes 集群。</p>
<p>文中清单和输出的节点名统一使用示例名称，复现时请替换为实际节点名。</p>
<table><thead><tr><th>项目</th><th>本次环境</th></tr></thead><tbody><tr><td>GPU Server</td><td><code>gpu-server-01</code>，8 vCPU、16 GiB 内存、Tesla T4 15360 MiB</td></tr><tr><td>CPU Client</td><td><code>cpu-client-01</code>，4 vCPU、约 8 GiB 内存，纯 CPU 节点</td></tr><tr><td>Kubernetes / containerd</td><td>v1.37.0 / 2.2.4</td></tr><tr><td>HAMi</td><td>v2.11.0</td></tr></tbody></table>
<p>CPU Client 上的 Pod 需要能够访问 GPU Server 的 InternalIP，默认服务端口为 TCP 14833。</p>
<p>查看集群节点，后续命令和 YAML 中的节点名需要替换为自己集群中的实际名称。</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl get nodes</span><br></div></code></pre></div></div>
<p>本次两个节点均为 Ready。</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">NAME            STATUS   ROLES           VERSION</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">cpu-client-01   Ready    &lt;none&gt;          v1.37.0</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">gpu-server-01   Ready    control-plane   v1.37.0</span><br></div></code></pre></div></div>
<p>本例的 GPU Server 同时是 master（控制平面）节点。如该节点存在控制平面的 <code>NoSchedule</code> 污点，部署前需先去除，以免 HAMi Device Plugin 和 Lupine Server 无法调度到该节点。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="32-部署-gpu-operator">3.2 部署 GPU Operator<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#32-%E9%83%A8%E7%BD%B2-gpu-operator" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>由 GPU Operator 安装 NVIDIA Driver、Container Toolkit 和监控组件，关闭原生 NVIDIA Device Plugin，后续由 HAMi 管理 GPU。</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm repo </span><span class="token function" style="color:rgb(130, 170, 255)">add</span><span class="token plain"> nvidia https://helm.ngc.nvidia.com/nvidia</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm repo update</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm upgrade </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--install</span><span class="token plain"> gpu-operator nvidia/gpu-operator </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> gpu-operator --create-namespace </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--version</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">v26.3.1 </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--set</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">driver.enabled</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">true </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--set</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">devicePlugin.enabled</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">false </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--wait</span><br></div></code></pre></div></div>
<blockquote>
<p>如果节点已经预装 NVIDIA Driver，可以把 <code>driver.enabled</code> 改为 <code>false</code>。</p>
</blockquote>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="33-安装-hami-并启用-remote-gpu">3.3 安装 HAMi 并启用 Remote GPU<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#33-%E5%AE%89%E8%A3%85-hami-%E5%B9%B6%E5%90%AF%E7%94%A8-remote-gpu" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm repo </span><span class="token function" style="color:rgb(130, 170, 255)">add</span><span class="token plain"> hami-charts https://project-hami.github.io/HAMi/</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm repo update hami-charts</span><br></div></code></pre></div></div>
<p>先安装 HAMi，并开启 Remote GPU：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl get runtimeclass</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm upgrade </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--install</span><span class="token plain"> hami hami-charts/hami </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--version</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">2.11</span><span class="token plain">.0 </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> hami-system --create-namespace </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--set</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">devices.remotegpu.enabled</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">true </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--set</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">devices.nvidia.runtimeClassName</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">nvidia </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--set</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">devices.remotegpu.server.runtimeClassName</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">nvidia </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--wait</span><br></div></code></pre></div></div>
<p>安装完成后，修改 Device Plugin 的节点配置，将 GPU Server 切换为 <code>remote</code> 模式：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> hami-system patch configmap hami-device-plugin </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--type</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">merge --patch-file</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">/dev/stdin </span><span class="token operator" style="color:rgb(137, 221, 255)">&lt;&lt;</span><span class="token string" style="color:rgb(195, 232, 141)">'EOF'</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">data:</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">  config.json: |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">    {</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">      "nodeconfig": [</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">        {</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">          "name": "gpu-server-01",</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">          "operatingmode": "remote"</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">        }</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">      ]</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">    }</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">EOF</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> hami-system rollout restart daemonset/hami-device-plugin</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> hami-system rollout status daemonset/hami-device-plugin </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--timeout</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">180s</span><br></div></code></pre></div></div>
<p>Device Plugin 会为 GPU Server 添加发现标签，远程服务端随之调度到该节点。</p>
<p>最后给节点打上 <code>gpu=on</code> 标签，未标记的节点不会被 HAMi 接管。</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># kubectl label nodes {nodeid} gpu=on</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl label nodes gpu-server-01 </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">gpu</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">on </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--overwrite</span><br></div></code></pre></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="34-检查部署结果">3.4 检查部署结果<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#34-%E6%A3%80%E6%9F%A5%E9%83%A8%E7%BD%B2%E7%BB%93%E6%9E%9C" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>检查 GPU Server 是否获得 <code>hami.io/lupine-server</code> 标签，以及 HAMi 各组件是否就绪。</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl get nodes </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-l</span><span class="token plain"> hami.io/lupine-server </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-o</span><span class="token plain"> wide</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> hami-system get pods </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-o</span><span class="token plain"> wide</span><br></div></code></pre></div></div>
<p>本次组件状态如下。</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">NAME                             READY   STATUS    NODE</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">hami-device-plugin-cphp2         2/2     Running   gpu-server-01</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">hami-lupine-server-4xjb8         1/1     Running   gpu-server-01</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">hami-scheduler-c9c5cf696-lgvgn   2/2     Running   cpu-client-01</span><br></div></code></pre></div></div>
<p>在远程服务容器内查询 GPU，确认服务端能够访问设备。</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> hami-system </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">exec</span><span class="token plain"> daemonset/hami-lupine-server -- </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  nvidia-smi --query-gpu</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">name,memory.total </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--format</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">csv,noheader</span><br></div></code></pre></div></div>
<p>本次输出如下。</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">Tesla T4, 15360 MiB</span><br></div></code></pre></div></div>
<p>到这里，GPU Server 已准备好。接下来在 CPU Client 上运行 CUDA Samples 和 vLLM，验证远程 GPU 的分配与实际计算。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="4-实战验证">4. 实战验证<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#4-%E5%AE%9E%E6%88%98%E9%AA%8C%E8%AF%81" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="41-先查看远程-gpu">4.1 先查看远程 GPU<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#41-%E5%85%88%E6%9F%A5%E7%9C%8B%E8%BF%9C%E7%A8%8B-gpu" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>先创建一个在 CPU 节点保持运行的 Pod，再通过 <code>kubectl exec</code> 执行 <code>nvidia-smi</code>，确认容器能看到远端 GPU。创建 <code>remote-gpu-smi.yaml</code>，将 <code>nodeSelector</code> 改为实际 CPU 节点名称。</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">apiVersion</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> v1</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">kind</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Pod</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">metadata</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> remote</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">smi</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">namespace</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> default</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">spec</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">nodeSelector</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">kubernetes.io/hostname</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> cpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">client</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token number" style="color:rgb(247, 140, 108)">01</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">restartPolicy</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Never</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">containers</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> smi</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">image</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm/vllm</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">openai</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">v0.25.1</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">ubuntu2404</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">command</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">[</span><span class="token string" style="color:rgb(195, 232, 141)">"sleep"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"infinity"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">]</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token key atrule">limits</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">nvidia.com/remote-gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><br></div></code></pre></div></div>
<p>创建 Pod，等待容器就绪后查看调度位置。</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl apply </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-f</span><span class="token plain"> remote-gpu-smi.yaml</span><br></div></code></pre></div></div>
<p>Pod 运行在 CPU 节点 <code>cpu-client-01</code>。</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">NAME             READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">remote-gpu-smi   1/1     Running   0          13s   172.25.58.225   cpu-client-01   &lt;none&gt;           &lt;none&gt;</span><br></div></code></pre></div></div>
<p>通过 <code>kubectl exec</code> 在容器内执行查询。</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">exec</span><span class="token plain"> remote-gpu-smi </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-c</span><span class="token plain"> smi -- nvidia-smi</span><br></div></code></pre></div></div>
<p><code>nvidia-smi</code> 完整输出如下。</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">Tue Nov 10 04:25:13 2026</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">+-----------------------------------------------------------------------------------------+</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| NVIDIA-SMI 615.71.09              KMD Version: 580.178.04    CUDA UMD Version: 13.3     |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">+-----------------------------------------+------------------------+----------------------+</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|                                         |                        |               MIG M. |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|=========================================+========================+======================|</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|   0  Tesla T4 (via lupine 172...    Off |   00000000:00:06.0 Off |                    0 |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| N/A   34C    P8             15W /   70W |       0MiB /  15360MiB |      0%      Default |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|                                         |                        |                  N/A |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">+-----------------------------------------+------------------------+----------------------+</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">+-----------------------------------------------------------------------------------------+</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| Processes:                                                                              |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|        ID   ID                                                               Usage      |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|=========================================================================================|</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|  No running processes found                                                             |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">+-----------------------------------------------------------------------------------------+</span><br></div></code></pre></div></div>
<p>可以看到远端 Tesla T4 和 15360 MiB 显存，设备名称中带有 <code>via lupine</code>。</p>
<p>Pod 通过 <code>sleep</code> 保持运行，仍保留远程 GPU 分配。查看完成后删除测试 Pod，再继续运行 CUDA 示例。</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl delete pod remote-gpu-smi</span><br></div></code></pre></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="42-运行-nvidia-官方-cuda-samples">4.2 运行 NVIDIA 官方 CUDA Samples<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#42-%E8%BF%90%E8%A1%8C-nvidia-%E5%AE%98%E6%96%B9-cuda-samples" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>在加载模型前，先独立验证 CUDA。这里选择 NVIDIA CUDA Samples 中的两个基础程序，分别检查设备发现和实际计算。</p>
<table><thead><tr><th>示例</th><th>验证内容</th><th>本次验收结果</th></tr></thead><tbody><tr><td><code>deviceQuery</code></td><td>通过 CUDA Runtime 查询设备数量、名称、计算能力和显存</td><td>发现 1 张 Tesla T4，计算能力 7.5，可见显存 3000 MiB，<code>Result = PASS</code></td></tr><tr><td><code>vectorAdd</code></td><td>分配显存、上传输入、执行向量加法 Kernel、回传结果并由 CPU 逐元素校验</td><td>50,000 个元素计算正确，<code>Test PASSED</code></td></tr></tbody></table>
<p><code>deviceQuery</code> 的无设备分支也可能输出 PASS，因此要同时核对设备数和卡名。清单增加了设备数量检查，随后由 <code>vectorAdd</code> 验证数据传输和 Kernel 的计算结果。</p>
<p>先在 CPU Client 上准备源码。固定到与镜像内 CUDA 13.0 对应的 <code>v13.0</code>。</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token function" style="color:rgb(130, 170, 255)">git</span><span class="token plain"> clone </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--depth</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><span class="token plain"> </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--branch</span><span class="token plain"> v13.0 </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--filter</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">blob:none </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--sparse</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  https://github.com/NVIDIA/cuda-samples.git /opt/cuda-samples-hami-v13.0</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token function" style="color:rgb(130, 170, 255)">git</span><span class="token plain"> </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-C</span><span class="token plain"> /opt/cuda-samples-hami-v13.0 sparse-checkout </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">set</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  Common Samples/1_Utilities/deviceQuery Samples/0_Introduction/vectorAdd</span><br></div></code></pre></div></div>
<p>创建 <code>cuda-samples-pod.yaml</code>，将 <code>nodeSelector</code> 改为实际 CPU 节点名称。</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">apiVersion</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> v1</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">kind</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Pod</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">metadata</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> cuda</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">samples</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">remote</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">namespace</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> default</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">spec</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">nodeSelector</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">kubernetes.io/hostname</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> cpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">client</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token number" style="color:rgb(247, 140, 108)">01</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">restartPolicy</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Never</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">containers</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> cuda</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">samples</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">image</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm/vllm</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">openai</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">v0.25.1</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">ubuntu2404</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">command</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">[</span><span class="token string" style="color:rgb(195, 232, 141)">"bash"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"-ec"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">]</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">args</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">|</span><span class="token scalar string" style="color:rgb(195, 232, 141)"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token scalar string" style="color:rgb(195, 232, 141)">          set -o pipefail</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token scalar string" style="color:rgb(195, 232, 141)">          NVCC=/usr/local/cuda-13.0/bin/nvcc</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token scalar string" style="color:rgb(195, 232, 141)">          cd /tmp</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          "$NVCC" </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">std=c++17 </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">O2 </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">arch=sm_75 </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">I/samples/Common \</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">            /samples/Samples/1_Utilities/deviceQuery/deviceQuery.cpp </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">o deviceQuery</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          "$NVCC" </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">std=c++17 </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">O2 </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">arch=sm_75 </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">I/samples/Common \</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">            /samples/Samples/0_Introduction/vectorAdd/vectorAdd.cu </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">o vectorAdd</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          ./deviceQuery </span><span class="token punctuation" style="color:rgb(199, 146, 234)">|</span><span class="token plain"> tee /tmp/device</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">query.log</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          grep </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">Eq 'Detected </span><span class="token punctuation" style="color:rgb(199, 146, 234)">[</span><span class="token plain">1</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token number" style="color:rgb(247, 140, 108)">9</span><span class="token punctuation" style="color:rgb(199, 146, 234)">]</span><span class="token punctuation" style="color:rgb(199, 146, 234)">[</span><span class="token plain">0</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token number" style="color:rgb(247, 140, 108)">9</span><span class="token punctuation" style="color:rgb(199, 146, 234)">]</span><span class="token plain">* CUDA Capable device' /tmp/device</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">query.log</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          ./vectorAdd</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          echo "REMOTE CUDA SAMPLES PASS"</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token key atrule">requests</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">cpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"1"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> 1Gi</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token key atrule">limits</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">cpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"2"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> 2Gi</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">nvidia.com/remote-gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">nvidia.com/remote-gpu-memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">3000</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">volumeMounts</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> samples</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">mountPath</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> /samples</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">readOnly</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token boolean important" style="color:rgb(255, 88, 116)">true</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">volumes</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> samples</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">hostPath</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token key atrule">path</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> /opt/cuda</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">samples</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">hami</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">v13.0</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token key atrule">type</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Directory</span><br></div></code></pre></div></div>
<p>在能访问集群的终端创建并等待执行完成。</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl apply </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-f</span><span class="token plain"> cuda-samples-pod.yaml</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl logs cuda-samples-remote </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-c</span><span class="token plain"> cuda-samples</span><br></div></code></pre></div></div>
<p>Pod 实际运行在纯 CPU 节点，输出如下：</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">Detected 1 CUDA Capable device(s)</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">Device 0: "Tesla T4"</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">Result = PASS</span><br></div></code></pre></div></div>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">[Vector addition of 50000 elements]</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">Test PASSED</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">Done</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">REMOTE CUDA SAMPLES PASS</span><br></div></code></pre></div></div>
<p>完成后删除 Pod，释放 Server 预留，再运行 vLLM。</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl delete pod cuda-samples-remote</span><br></div></code></pre></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="43-在-cpu-节点部署-vllm">4.3 在 CPU 节点部署 vLLM<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#43-%E5%9C%A8-cpu-%E8%8A%82%E7%82%B9%E9%83%A8%E7%BD%B2-vllm" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<blockquote>
<p>先在 CPU Client 上准备好 <code>Qwen2.5-0.5B-Instruct</code> 模型，存放到 <code>/opt/models/Qwen2.5-0.5B-Instruct</code> 目录。</p>
</blockquote>
<p>创建 <code>vllm-remote.yaml</code>。</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">apiVersion</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> v1</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">kind</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Pod</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">metadata</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">remote</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">spec</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">nodeSelector</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">kubernetes.io/hostname</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> cpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">client</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token number" style="color:rgb(247, 140, 108)">01</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">containers</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">image</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm/vllm</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">openai</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">v0.25.1</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">ubuntu2404</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">command</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">[</span><span class="token string" style="color:rgb(195, 232, 141)">"vllm"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"serve"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"/models/Qwen2.5-0.5B-Instruct"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">]</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">args</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">served</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">model</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">name=qwen</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">dtype=float16</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">utilization=0.7</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">max</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">model</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">len=1024</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">max</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">num</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">seqs=4</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">enforce</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">eager</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">attention</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">config.backend=TRITON_ATTN</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">env</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> TORCH_BLAS_PREFER_CUBLASLT</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">value</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"0"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> CUDA_LAUNCH_BLOCKING</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">value</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"1"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">readinessProbe</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token key atrule">httpGet</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">path</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> /health</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">port</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">8000</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token key atrule">limits</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">nvidia.com/remote-gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">volumeMounts</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> models</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">mountPath</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> /models</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">readOnly</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token boolean important" style="color:rgb(255, 88, 116)">true</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">volumes</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> models</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">hostPath</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token key atrule">path</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> /opt/models</span><br></div></code></pre></div></div>
<p>Pod 申请 1 张远程 GPU，<code>gpu-memory-utilization=0.7</code> 控制 vLLM 的显存使用比例。</p>
<p><code>--enforce-eager</code> 和 <code>CUDA_LAUNCH_BLOCKING=1</code> 沿用官方示例。</p>
<p>创建 Pod，等待服务就绪，再查看调度位置和启动日志。</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl apply </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-f</span><span class="token plain"> vllm-remote.yaml</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl get pod vllm-remote </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-o</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">jsonpath</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token string" style="color:rgb(195, 232, 141)">'{.spec.nodeName}{"\n"}{.metadata.annotations.hami\.io/lupine-endpoint}{"\n"}'</span><br></div></code></pre></div></div>
<p>本次分配结果如下。</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">cpu-client-01</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">172.16.131.132:14833</span><br></div></code></pre></div></div>
<p>Pod 进入 Running 后，继续等待模型加载完成，以 <code>/health</code> 返回 200 作为服务就绪的依据。在 GPU Server 的宿主机执行 <code>nvidia-smi</code>，查看 GPU 显存占用和进程信息。</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">nvidia-smi</span><br></div></code></pre></div></div>
<p>本次模型加载完成后的完整输出如下。</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">Tue Nov 10 05:55:16 2026</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">+-----------------------------------------------------------------------------------------+</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| NVIDIA-SMI 580.178.04             Driver Version: 580.178.04     CUDA Version: 13.0     |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">+-----------------------------------------+------------------------+----------------------+</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|                                         |                        |               MIG M. |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|=========================================+========================+======================|</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|   0  Tesla T4                       Off |   00000000:00:06.0 Off |                    0 |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| N/A   41C    P0             28W /   70W |   10765MiB /  15360MiB |      0%      Default |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|                                         |                        |                  N/A |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">+-----------------------------------------+------------------------+----------------------+</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">+-----------------------------------------------------------------------------------------+</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| Processes:                                                                              |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|        ID   ID                                                               Usage      |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|=========================================================================================|</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|    0   N/A  N/A         3442254      C   ...pine/bin/lupine_driver_server        122MiB |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">|    0   N/A  N/A         3443585      C   ...pine/bin/lupine_driver_server      10640MiB |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">+-----------------------------------------------------------------------------------------+</span><br></div></code></pre></div></div>
<p>可以看到，T4 的显存占用已增加到 10765 MiB。进程列表中的 <code>lupine_driver_server</code> 是 GPU Server 上的远程服务进程，负责执行 GPU 计算。vLLM 应用进程则运行在 CPU Client 上。</p>
<p>在能访问集群 Pod 网络的机器上，通过 OpenAI 兼容接口发起请求。本次从 GPU 节点请求 CPU 节点上的 vLLM 服务。</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token assign-left variable" style="color:rgb(191, 199, 213)">POD_IP</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token variable" style="color:rgb(191, 199, 213)">$(</span><span class="token variable" style="color:rgb(191, 199, 213)">kubectl get pod vllm-remote </span><span class="token variable punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token variable" style="color:rgb(191, 199, 213)"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token variable" style="color:rgb(191, 199, 213)">  </span><span class="token variable parameter variable" style="color:rgb(191, 199, 213)">-o</span><span class="token variable" style="color:rgb(191, 199, 213)"> </span><span class="token variable assign-left variable" style="color:rgb(191, 199, 213)">jsonpath</span><span class="token variable operator" style="color:rgb(137, 221, 255)">=</span><span class="token variable string" style="color:rgb(195, 232, 141)">'{.status.podIP}'</span><span class="token variable" style="color:rgb(191, 199, 213)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token function" style="color:rgb(130, 170, 255)">curl</span><span class="token plain"> </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--fail</span><span class="token plain"> </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--silent</span><span class="token plain"> --show-error </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-o</span><span class="token plain"> /dev/null </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-w</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">'%{http_code}\n'</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token string" style="color:rgb(195, 232, 141)">"http://</span><span class="token string variable" style="color:rgb(191, 199, 213)">${POD_IP}</span><span class="token string" style="color:rgb(195, 232, 141)">:8000/health"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token function" style="color:rgb(130, 170, 255)">curl</span><span class="token plain"> --fail-with-body </span><span class="token string" style="color:rgb(195, 232, 141)">"http://</span><span class="token string variable" style="color:rgb(191, 199, 213)">${POD_IP}</span><span class="token string" style="color:rgb(195, 232, 141)">:8000/v1/chat/completions"</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-H</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">'Content-Type: application/json'</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-d</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">'{"model":"qwen","messages":[{"role":"user","content":"用一句话解释 Kubernetes。"}],"max_tokens":64,"temperature":0}'</span><br></div></code></pre></div></div>
<p>输出如下：</p>
<blockquote>
<p>Kubernetes（简称K8s）是一种开源的容器编排平台，它允许用户定义和管理应用程序在多个节点上的部署、运行和扩展。通过使用Kubernetes，开发者可以轻松地创建、管理和维护复杂的容器化应用集群，从而提高开发效率和资源利用率。</p>
</blockquote>
<p>至此，我们使用 Remote GPU 成功完成了一次推理请求。</p>
<p>完成实验后，删除测试 Pod，释放远程 GPU 资源。</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl delete pod vllm-remote</span><br></div></code></pre></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="5-使用场景与挑战">5. 使用场景与挑战<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#5-%E4%BD%BF%E7%94%A8%E5%9C%BA%E6%99%AF%E4%B8%8E%E6%8C%91%E6%88%98" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="51-应用与-gpu-分开部署有什么价值">5.1 应用与 GPU 分开部署，有什么价值<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#51-%E5%BA%94%E7%94%A8%E4%B8%8E-gpu-%E5%88%86%E5%BC%80%E9%83%A8%E7%BD%B2%E6%9C%89%E4%BB%80%E4%B9%88%E4%BB%B7%E5%80%BC" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>Lupine 将自己定义为 GPU-over-IP bridge，让 CPU-only 机器能够通过网络访问其他机器上的 GPU。HAMi 则进一步明确了 Kubernetes 中的运行方式，工作负载运行在 Client 节点，CUDA 调用在另一台 GPU Server 上执行，由 HAMi 完成 Server 选择和客户端库下发。</p>
<p>从资源部署的角度看，可以将它理解为 CPU 与 GPU 的解耦，将 GPU 作为一种特殊资源统一管理。应用使用哪台机器的 CPU、内存，与使用哪台机器的 GPU，可以分别安排。</p>
<p>Remote GPU 的价值在于提供另一种部署选择，是否有收益还需要结合业务验证。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="52-哪些场景值得尝试">5.2 哪些场景值得尝试<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#52-%E5%93%AA%E4%BA%9B%E5%9C%BA%E6%99%AF%E5%80%BC%E5%BE%97%E5%B0%9D%E8%AF%95" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>结合这种执行方式，可以考虑以下几类场景。</p>
<p><strong>应用需要较多 CPU 或内存。</strong> 数据预处理、后处理与 GPU 计算对资源的需求可能不同。将应用部署在 CPU、内存规格合适的节点，让 GPU Server 提供计算设备，可以分别选择两侧的资源配置。不过，数据传输成本也需要一起评估。GPU API Remoting 的<a href="https://arxiv.org/html/2401.13354v1#S2.SS2" target="_blank" rel="noopener noreferrer" class="">研究论文</a>也将 CPU/GPU 固定配比与应用需求不匹配，列为资源解耦的动机之一。</p>
<p><strong>GPU 集中部署，应用分散运行。</strong> 将 GPU 和驱动集中维护在 Server 侧，让不同 CPU 节点上的应用按需访问，是远程 GPU 的另一种部署思路。在 HAMi 中，可以先从集群内的 CPU Client 与 GPU Server 分离开始验证；集中部署后的共享、隔离和调度能力，需要按具体实现分别确认。</p>
<p><strong>开发与功能测试。</strong> 无 GPU 的测试节点可以通过远端 GPU 验证 CUDA 程序、框架和模型的运行流程。本文依次使用 <code>nvidia-smi</code>、CUDA Samples 和 vLLM，分别验证设备可见性、实际 CUDA 计算和推理服务响应，这套顺序也适合用来检查环境。例如让 CPU-only Runner 连接远端 GPU，执行需要 GPU 的工作负载。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="53-使用前需要关注什么">5.3 使用前需要关注什么<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#53-%E4%BD%BF%E7%94%A8%E5%89%8D%E9%9C%80%E8%A6%81%E5%85%B3%E6%B3%A8%E4%BB%80%E4%B9%88" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p><strong>性能取决于网络，也取决于远程调用的实现。</strong> Client 与 Server 之间传输的是 CUDA 调用及相关数据，频繁同步会放大往返时延，大量数据搬运则消耗带宽。此外，参数序列化、反序列化和调用转发也有软件开销。<a href="https://arxiv.org/html/2401.13354v1#S3" target="_blank" rel="noopener noreferrer" class="">GPU API Remoting 研究</a>专门分析了这些成本，以及异步执行对开销的影响。因此，不能只用网络带宽判断性能，也不能将吞吐下降全部归因于网络。本文验证了功能，业务延迟与吞吐仍需实测。</p>
<p><strong>客户端无需 NVIDIA 驱动，仍需兼容的运行环境。</strong> HAMi 官方说明，当前自动库下发面向提供的 amd64、Ubuntu 24.04 CUDA 和 vLLM 示例镜像。自定义镜像可能需要适配库搜索路径，通过绝对路径加载的库也需要单独检查。因此，更换 CUDA、框架或镜像版本后，应重新执行基础计算和应用验证。</p>
<p><strong>故障排查需要覆盖两端。</strong> 应用启动失败或推理中断时，需要同时检查 Client、网络、远程服务和 GPU Server。连接中断、Server 重启后的恢复方式，以及长时间运行的稳定性，都应纳入上线前的验证。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="6-小结">6. 小结<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#6-%E5%B0%8F%E7%BB%93" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>本文通过 HAMi v2.11 的 Remote GPU，在没有 GPU 和 NVIDIA 驱动的 CPU 节点上，跑通了 CUDA Samples 和 vLLM 推理。应用进程运行在 CPU Client，由另一节点上的 T4 执行 GPU 计算，完成了从远程设备发现、CUDA 计算到模型推理的完整验证。</p>
<p>Remote GPU 为 GPU 工作负载增加了一种部署选择。应用可以使用 CPU 节点上的计算和内存资源，同时通过网络调用 GPU Server 的算力。HAMi 将远程 GPU 的发现、分配和客户端环境注入接入已有的调度流程，让这套跨节点使用 GPU 的方式可以通过熟悉的 Pod 资源声明来管理。</p>
<p>对于希望解耦 CPU 与 GPU 部署、集中提供 GPU 计算能力的平台而言，HAMi 提供了一个值得尝试的起点，也让 AI 基础设施的资源组合有了更多可能。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="延伸阅读">延伸阅读<a href="https://project-hami.io/zh/blog/hami-v2-11-remote-gpu#%E5%BB%B6%E4%BC%B8%E9%98%85%E8%AF%BB" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class=""><a class="" href="https://project-hami.io/zh/blog/hami-v2-11-0-release">HAMi v2.11.0 发布介绍</a></li>
<li class=""><a class="" href="https://project-hami.io/zh/docs/userguide/nvidia-device/remote-gpu">Remote GPU 用户指南</a></li>
<li class=""><a href="https://github.com/Project-HAMi/HAMi/tree/v2.11.0/examples/remotegpu" target="_blank" rel="noopener noreferrer" class="">HAMi v2.11.0 Remote GPU 官方示例</a></li>
<li class=""><a href="https://arxiv.org/abs/2401.13354" target="_blank" rel="noopener noreferrer" class="">Characterizing Network Requirements for GPU API Remoting in AI Applications</a></li>
</ul>]]></content>
        <author>
            <name>意琦行</name>
            <uri>https://github.com/lixd</uri>
        </author>
        <category label="HAMi" term="HAMi"/>
        <category label="GPU" term="GPU"/>
        <category label="Kubernetes" term="Kubernetes"/>
        <category label="vLLM" term="vLLM"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[HAMi 加入 Vultr 开源联盟]]></title>
        <id>https://project-hami.io/zh/blog/hami-joins-vultr-open-source-alliance</id>
        <link href="https://project-hami.io/zh/blog/hami-joins-vultr-open-source-alliance"/>
        <updated>2026-11-11T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[HAMi 宣布加入新成立的 Vultr 开源联盟，成为六家创始成员之一，与 Spinifex、Upbound、Qdrant、Valkey、NetBird 一道，共同推进开放、可企业化落地的 AI 基础设施。]]></summary>
        <content type="html"><![CDATA[<p>HAMi 社区高兴地宣布，HAMi 已加入新成立的 <strong>Vultr 开源联盟</strong>（Vultr Open Source Alliance），成为其六家创始成员之一，其他创始成员还包括 Spinifex、Upbound、Qdrant、Valkey 与 NetBird。</p>
<p><a href="https://blogs.vultr.com/vultr-open-source-alliance" target="_blank" rel="noopener noreferrer" class="">该联盟于 2026 年 11 月 10 日正式发布</a>，汇聚开源项目与基础设施提供商，帮助组织以生产级云基础设施的性能、规模与可控性来部署和运营开源技术。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="让-gpu-基础设施更加开放与高效">让 GPU 基础设施更加开放与高效<a href="https://project-hami.io/zh/blog/hami-joins-vultr-open-source-alliance#%E8%AE%A9-gpu-%E5%9F%BA%E7%A1%80%E8%AE%BE%E6%96%BD%E6%9B%B4%E5%8A%A0%E5%BC%80%E6%94%BE%E4%B8%8E%E9%AB%98%E6%95%88" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>随着 AI 工作负载持续增长，高效的 GPU 资源管理对构建和运营 AI 基础设施的组织而言愈发重要。</p>
<p>HAMi 是 CNCF 孵化阶段项目，通过在 Kubernetes 环境中实现 GPU 共享、资源隔离与异构算力调度来应对这一挑战。</p>
<p>通过允许多个工作负载在保持资源边界的前提下共享 GPU 资源，HAMi 帮助组织提升利用率，更充分地使用现有算力基础设施。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="vultr-开源联盟的意义">Vultr 开源联盟的意义<a href="https://project-hami.io/zh/blog/hami-joins-vultr-open-source-alliance#vultr-%E5%BC%80%E6%BA%90%E8%81%94%E7%9B%9F%E7%9A%84%E6%84%8F%E4%B9%89" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>Vultr 开源联盟将开源技术与云基础设施相结合，降低企业采用开源技术的门槛。</p>
<p>对 HAMi 而言，这为探索开源 GPU 资源管理与云 GPU 基础设施、Kubernetes 环境的结合创造了机会。</p>
<p>Vultr 的 Cloud GPU 与托管 Kubernetes 服务为探索面向 AI 应用的 GPU 共享、工作负载隔离与资源优化提供了合适的基础。</p>
<p>从更大的范围看，这次合作体现了双方对开放、可互操作、可移植基础设施的共同承诺，为开发者带来更大的灵活性与掌控力。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="感谢-vultr-对-hami-社区的支持">感谢 Vultr 对 HAMi 社区的支持<a href="https://project-hami.io/zh/blog/hami-joins-vultr-open-source-alliance#%E6%84%9F%E8%B0%A2-vultr-%E5%AF%B9-hami-%E7%A4%BE%E5%8C%BA%E7%9A%84%E6%94%AF%E6%8C%81" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>感谢 Vultr 长期以来对 HAMi 社区的支持。</p>
<p>Vultr 为社区提供了测试环境。在 <a class="" href="https://project-hami.io/zh/blog/hami-v2-11-0-release">HAMi v2.11.0</a> 发布期间，社区借助 Vultr 的云 GPU 测试环境完成了多项新能力的验证。这类支持让社区能够在真实基础设施上开展测试，也让开源开发与生产环境保持贴近。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="展望">展望<a href="https://project-hami.io/zh/blog/hami-joins-vultr-open-source-alliance#%E5%B1%95%E6%9C%9B" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>作为联盟的创始成员，HAMi 社区期待在技术协作上展开探索，包括部署指南、参考架构，以及帮助开发者评估和采用 GPU 共享技术的实用资源。</p>
<p>我们的目标没有改变：通过开源，让异构算力更易获得、更高效、更易管理。</p>
<p>我们欢迎与基础设施提供商、开发者以及更广泛的云原生社区协作，共同推进开放的 AI 基础设施。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="了解更多">了解更多<a href="https://project-hami.io/zh/blog/hami-joins-vultr-open-source-alliance#%E4%BA%86%E8%A7%A3%E6%9B%B4%E5%A4%9A" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class=""><a href="https://blogs.vultr.com/vultr-open-source-alliance" target="_blank" rel="noopener noreferrer" class="">Vultr 开源联盟发布公告</a>（英文）</li>
<li class=""><a href="https://project-hami.io/" target="_blank" rel="noopener noreferrer" class="">HAMi 官方网站</a></li>
<li class=""><a href="https://github.com/Project-HAMi/HAMi" target="_blank" rel="noopener noreferrer" class="">HAMi GitHub 仓库</a></li>
</ul>]]></content>
        <author>
            <name>HAMi 社区</name>
        </author>
        <category label="HAMi" term="HAMi"/>
        <category label="GPU" term="GPU"/>
        <category label="Kubernetes" term="Kubernetes"/>
        <category label="CNCF" term="CNCF"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[HAMi v2.11.0 发布：Remote GPU 调度、AMD GPU 共享、昇腾 ENPU、NUMA Refit 与走向成熟的 Flexible MIG]]></title>
        <id>https://project-hami.io/zh/blog/hami-v2-11-0-release</id>
        <link href="https://project-hami.io/zh/blog/hami-v2-11-0-release"/>
        <updated>2026-11-04T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[HAMi v2.11.0 正式发布。本次发布带来基于 Lupine 的 Remote GPU 调度、具备容器级显存与算力限制的完整 AMD Instinct GPU 共享、支持策略隔离的昇腾 ENPU 模式、对齐 CPU 与 GPU 调度的 NUMA refit、包括 CDI 支持在内的 Flexible MIG 重要完善、调度器安全加固、OpenShift 与 SELinux 支持，以及重构后的 Helm Chart。]]></summary>
        <content type="html"><![CDATA[<p>HAMi 社区正式发布 <strong>HAMi v2.11.0</strong>。</p>
<p>本次发布要点：</p>
<ul>
<li class=""><strong>Remote GPU</strong>：在没有 GPU、没有 NVIDIA 驱动的客户端节点上运行 CUDA 与 vLLM Pod，使用 <a href="https://github.com/lupinemachines" target="_blank" rel="noopener noreferrer" class="">Lupine</a> GPU 服务器通过网络提供的 GPU。</li>
<li class=""><strong>AMD Instinct GPU 共享</strong>：AMD GPU（ROCm）成为 HAMi 的一等公民，支持按大小或百分比限制显存、限制算力、NUMA 与 XGMI 感知放置、调度策略与队列优先级，全部在容器级强制执行。</li>
<li class=""><strong>昇腾 ENPU 模式</strong>：与模板模式、hami-core 并列的第三种切分方式，提供 fixed-share、elastic、best-effort 策略隔离，同时 <code>hami-vnpu-core</code> 完成大规模性能优化。</li>
<li class=""><strong>NUMA refit</strong>：kubelet Topology Manager 与 HAMi 调度器达成一致，CPU 亲和性与 GPU 放置落在同一 NUMA 节点。</li>
<li class=""><strong>Flexible MIG 走向成熟</strong>：动态 MIG 设备支持 CDI 注入、带回退的放置打分、按 Pod 的 profile 偏好。</li>
<li class=""><strong>更安全的调度器</strong>：安全审计驱动下的 extender 与 webhook 加固，Filter 路径性能提升。</li>
<li class=""><strong>平台与 Chart</strong>：OpenShift 与 SELinux 支持、重构后的 <code>devices.VENDOR</code> chart、节点容量资源上报。</li>
<li class=""><strong>官网大改版</strong>：信息架构重组、Concepts 章节重绘、<code>llms.txt</code>、安装与升级指南重构。</li>
</ul>
<p>v2.11.0 共合并 220 多个 pull request，来自 50 多位贡献者，其中 <strong>29 位是首次贡献</strong>。</p>
<p>本文将对 v2.11.0 的主要更新进行详细说明。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="remote-gpu在任何地方运行-cuda">Remote GPU：在任何地方运行 CUDA<a href="https://project-hami.io/zh/blog/hami-v2-11-0-release#remote-gpu%E5%9C%A8%E4%BB%BB%E4%BD%95%E5%9C%B0%E6%96%B9%E8%BF%90%E8%A1%8C-cuda" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>不是每个节点都需要 GPU，也不是每个 GPU 节点都应该跑工作负载。借助 Remote GPU（<a href="https://github.com/Project-HAMi/HAMi/pull/3004" target="_blank" rel="noopener noreferrer" class="">#3004</a>，<a href="https://github.com/moezdil" target="_blank" rel="noopener noreferrer" class="">@moezdil</a>；设计方案来自 <a href="https://github.com/lupinemachines" target="_blank" rel="noopener noreferrer" class="">Lupine</a> 项目），HAMi 现在可以把 Pod 调度到由 Lupine GPU 服务器通过网络提供的 GPU 上：</p>
<ul>
<li class=""><strong>无驱动的客户端节点</strong>：客户端节点不需要 GPU，也不需要 NVIDIA 驱动。HAMi 把 Pod 放置到独立的客户端节点，通过 init 容器下发 Lupine 客户端库和 <code>nvidia-smi</code>，并注入指向所选服务器的 <code>LUPINE_SERVER</code> 环境变量（<a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">@archlitchi</a>）。</li>
<li class=""><strong>GPU 节点变身 Lupine 服务器</strong>：配置了 <code>operatingmode: remote</code> 的节点不再把 GPU 作为本地设备上报，而是为 Lupine 运行 GPU，并被打上 <code>hami.io/lupine-server</code> 标签（<a href="https://github.com/Project-HAMi/HAMi/pull/3013" target="_blank" rel="noopener noreferrer" class="">#3013</a>，<a href="https://github.com/moezdil" target="_blank" rel="noopener noreferrer" class="">@moezdil</a>）。</li>
<li class=""><strong>资源池选择</strong>：Lupine 服务器可以按节点标签划分成多个池，Pod 通过 <code>hami.io/remote-server</code> 注解请求特定池（<a href="https://github.com/Project-HAMi/HAMi/pull/3195" target="_blank" rel="noopener noreferrer" class="">#3195</a>，<a href="https://github.com/asadjan4611" target="_blank" rel="noopener noreferrer" class="">@asadjan4611</a>）。NotReady 的服务器会被排除在放置之外。</li>
<li class=""><strong>整机预留 + 每 GPU 显存限制</strong>：<code>nvidia.com/remote-gpu</code> 预留整个服务器（GPU 数量不少于请求值），可选的 <code>nvidia.com/remote-gpu-memory</code> 设置每 GPU 显存上限，由注入的 HAMi-core 库强制执行。多容器 Pod 保持在同一台服务器上（<a href="https://github.com/sanket-jadhav-cse" target="_blank" rel="noopener noreferrer" class="">@sanket-jadhav-cse</a>）。</li>
<li class=""><strong>运维安全</strong>：等待 Lupine 集群变化的 Pod 会被重新入队而不是直接失败，Lupine 服务器支持可配置的 securityContext，可选的会话中继让用户无需服务器命名空间的转发权限即可 <code>kubectl port-forward</code>。<a href="https://github.com/FouoF" target="_blank" rel="noopener noreferrer" class="">@FouoF</a> 对 remote 模式做了进一步加固。</li>
</ul>
<p><img decoding="async" loading="lazy" alt="Remote GPU（Lupine）架构" src="https://project-hami.io/zh/assets/images/remote-gpu-architecture-bdb831e8d0d4c9e5bb2206f7e06f225d.svg#light-only" width="1101" height="761" class="img_ev3q"> <img decoding="async" loading="lazy" alt="Remote GPU（Lupine）架构" src="https://project-hami.io/zh/assets/images/remote-gpu-architecture-dark-3f2dd6a74bd7a13f6b648e496811827e.svg#dark-only" width="1101" height="761" class="img_ev3q"></p>
<p><strong>启用方式</strong>只需要两个 chart 配置和一处节点配置：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">devices</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">remotegpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">enabled</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token boolean important" style="color:rgb(255, 88, 116)">true</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">defaultPort</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">14833</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">resourceCountName</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"nvidia.com/remote-gpu"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">resourceMemoryName</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"nvidia.com/remote-gpu-memory"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">devicePlugin</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">nodeConfiguration</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">config</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">|</span><span class="token scalar string" style="color:rgb(195, 232, 141)"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token scalar string" style="color:rgb(195, 232, 141)">      {</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token scalar string" style="color:rgb(195, 232, 141)">        "nodeconfig": [</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token scalar string" style="color:rgb(195, 232, 141)">          {</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token scalar string" style="color:rgb(195, 232, 141)">            "nodelabelselector": { "matchLabels": { "role": "lupine-server" } },</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token scalar string" style="color:rgb(195, 232, 141)">            "operatingmode": "remote"</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token scalar string" style="color:rgb(195, 232, 141)">          }</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token scalar string" style="color:rgb(195, 232, 141)">        ]</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token scalar string" style="color:rgb(195, 232, 141)">      }</span><br></div></code></pre></div></div>
<p>上面的 <code>nodelabelselector</code> 匹配的是你自行打上的标签；请在启用 remote 模式前（或之后重启 device-plugin）给每个要充当 Lupine 服务器的 GPU 节点打标签。该节点还必须匹配 <code>devicePlugin.nvidiaNodeSelector</code>（默认 <code>gpu=on</code>），否则 device plugin 与 Lupine 服务器不会被调度到该节点：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl label </span><span class="token function" style="color:rgb(130, 170, 255)">node</span><span class="token plain"> </span><span class="token operator" style="color:rgb(137, 221, 255)">&lt;</span><span class="token plain">gpu-node</span><span class="token operator" style="color:rgb(137, 221, 255)">&gt;</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">gpu</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">on </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">role</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">lupine-server</span><br></div></code></pre></div></div>
<p>之后，在无 GPU 节点上的 vLLM Deployment 看起来和普通 HAMi 工作负载完全一样（完整示例见 <a href="https://github.com/Project-HAMi/HAMi/tree/master/examples/remotegpu" target="_blank" rel="noopener noreferrer" class="">remote-gpu examples</a>）：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">apiVersion</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> apps/v1</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">kind</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Deployment</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">metadata</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">remote</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">gpu</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">spec</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">selector</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">matchLabels</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">app</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">remote</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">gpu</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">template</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">metadata</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">labels</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token key atrule">app</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">remote</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">gpu</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">spec</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">containers</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">image</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm/vllm</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">openai</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">v0.25.1</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">ubuntu2404</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">command</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">[</span><span class="token string" style="color:rgb(195, 232, 141)">"vllm"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"serve"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"Qwen/Qwen3-8B"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">]</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">            </span><span class="token key atrule">limits</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">              </span><span class="token key atrule">memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> 32Gi</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">              </span><span class="token key atrule">nvidia.com/remote-gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">2</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">              </span><span class="token key atrule">nvidia.com/remote-gpu-memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">20480</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 每 GPU 20 GiB</span><br></div></code></pre></div></div>
<p>当前需要了解的限制：客户端节点必须与服务器节点分开部署；支持的镜像为 CUDA 和 vLLM 0.19-0.26；工作负载会预留整个服务器。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="amd-instinct-gpu-共享">AMD Instinct GPU 共享<a href="https://project-hami.io/zh/blog/hami-v2-11-0-release#amd-instinct-gpu-%E5%85%B1%E4%BA%AB" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>HAMi 现在可以在 Kubernetes 上共享并调度 AMD Instinct GPU（ROCm），使用与 NVIDIA 相同的资源模型，并在容器内部强制执行硬限制。v2.10.0 打下了显存、算力、类型与 UUID 的基础；v2.11.0 补齐了完整的特性集（主要由 <a href="https://github.com/moezdil" target="_blank" rel="noopener noreferrer" class="">@moezdil</a> 完成）：</p>
<ul>
<li class=""><strong>显存隔离</strong>：通过 <code>amd.com/gpumem</code> 申请绝对大小，或通过 <code>amd.com/gpumem-percentage</code> 申请 VRAM 的百分比（<a href="https://github.com/Project-HAMi/HAMi/pull/3179" target="_blank" rel="noopener noreferrer" class="">#3179</a>）。限制由 <a href="https://github.com/Project-HAMi/amd-hami-core" target="_blank" rel="noopener noreferrer" class="">amd-hami-core</a> 执行，它现在还会对 <code>hipMalloc3D</code>、数组、内存池与 VMM 封顶，把限额作为设备总量上报，并在设置 <code>HIP_OVERSUBSCRIBE</code> 时用托管内存提供 <code>hipMalloc</code>。</li>
<li class=""><strong>算力隔离</strong>：<code>amd.com/gpucores</code> 通过 <code>HSA_CU_MASK</code> 设置计算单元份额，RDNA 上的算力请求会向上取整到整个 WGP（<a href="https://github.com/Project-HAMi/HAMi/pull/3174" target="_blank" rel="noopener noreferrer" class="">#3174</a>）。</li>
<li class=""><strong>灵活放置</strong>：选择或排除 GPU 类型与 UUID（<code>amd.com/use-gputype</code>、<code>amd.com/use-gpu-uuid</code> 及对应的 <code>nouse-</code> 注解），用 <code>amd.com/numa-bind</code> 把多 GPU Pod 固定在同一 NUMA 节点（<a href="https://github.com/Project-HAMi/HAMi/pull/3186" target="_blank" rel="noopener noreferrer" class="">#3186</a>），用 <code>amd.com/link-bind</code> 要求所有 GPU 通过 XGMI 互联，<code>topology-aware</code> 策略则优先选择 XGMI 链路最多的组合（<a href="https://github.com/Project-HAMi/HAMi/pull/3191" target="_blank" rel="noopener noreferrer" class="">#3191</a>）。</li>
<li class=""><strong>高级调度</strong>：支持 <code>binpack</code>、<code>spread</code>、<code>mutex</code> 与 <code>topology-aware</code> 策略，通过 <code>AMD_TASK_PRIORITY</code> 设置 Pod 队列优先级，在 Fit 阶段检查命名空间 ResourceQuota（<a href="https://github.com/Project-HAMi/HAMi/pull/3180" target="_blank" rel="noopener noreferrer" class="">#3180</a>），拓扑调度中只选择满足配额的 GPU 组合（<a href="https://github.com/Project-HAMi/HAMi/pull/3361" target="_blank" rel="noopener noreferrer" class="">#3361</a>，<a href="https://github.com/Eshiv-Pandey" target="_blank" rel="noopener noreferrer" class="">@Eshiv-Pandey</a>），为 GPU 数量、显存与算力提供默认值并支持设备封锁（<a href="https://github.com/Project-HAMi/HAMi/pull/3192" target="_blank" rel="noopener noreferrer" class="">#3192</a>），可配置 <code>runtimeClassName</code>（<a href="https://github.com/Project-HAMi/HAMi/pull/3194" target="_blank" rel="noopener noreferrer" class="">#3194</a>），按可分配 GPU 判断节点健康，以及专门的 AMD e2e 测试套件（<a href="https://github.com/Project-HAMi/HAMi/pull/3193" target="_blank" rel="noopener noreferrer" class="">#3193</a>）。</li>
</ul>
<p>节点侧，<a href="https://github.com/Project-HAMi/amd-device-plugin" target="_blank" rel="noopener noreferrer" class="">amd-device-plugin</a> v0.0.4 新增 Instinct GPU 的硬件算力分区（SPX、DPX、QPX、CPX）、来自 AMD Device Metrics Exporter 的单 GPU 健康状态、节点注解中的 XGMI 对端信息，以及用于显存超售的 <code>-device_memory_scaling</code>。全新的 <a href="https://github.com/Project-HAMi/amd-dra-driver" target="_blank" rel="noopener noreferrer" class="">amd-dra-driver</a>（v0.1.0 至 v0.4.1）把同样的限制带到 DRA 路径，为每张 GPU 单独下发 <code>HSA_CU_MASK</code> 与 <code>HIP_DEVICE_MEMORY_LIMIT_</code> 并向 kubelet 上报健康状态，<a href="https://github.com/Project-HAMi/HAMi-dra" target="_blank" rel="noopener noreferrer" class="">HAMi-dra</a> 则把 <code>amd.com/gpu</code> 请求转换为它的 claim。NVIDIA 与 AMD 由同一个调度层统一纳管，HAMi 为异构集群提供了统一的 GPU 共享能力；请参阅 <a class="" href="https://project-hami.io/zh/docs/next/userguide/amd-device/enable-amd-gpu-sharing">AMD GPU 指南</a>与<a class="" href="https://project-hami.io/zh/docs/next/userguide/amd-device/verification-status">验证状态</a>了解在真实硬件上验证过的范围。</p>
<p>AMD 支持是社区协作的成果：<a href="https://github.com/moezdil" target="_blank" rel="noopener noreferrer" class="">@moezdil</a> 在 Claven 的协助下打磨了调度器、device plugin、amd-hami-core 与 DRA driver 中的 Instinct 支持；初始设备支持由 <a href="https://github.com/FouoF" target="_blank" rel="noopener noreferrer" class="">@FouoF</a> 在 DigitalOcean 提供的 AMD 实例上实现；算力分区（SPX、DPX、QPX、CPX）与显存分区的初始工作由 <a href="https://github.com/fishman" target="_blank" rel="noopener noreferrer" class="">@fishman</a> 完成。特别感谢 <strong><a href="https://www.vultr.com/" target="_blank" rel="noopener noreferrer" class="">Vultr</a></strong> 提供 AMD 测试机器，也感谢 AMD 社区让这一切成为可能。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="更深入的昇腾切分enpu-模式与更快的-hami-core">更深入的昇腾切分：ENPU 模式与更快的 HAMi-core<a href="https://project-hami.io/zh/blog/hami-v2-11-0-release#%E6%9B%B4%E6%B7%B1%E5%85%A5%E7%9A%84%E6%98%87%E8%85%BE%E5%88%87%E5%88%86enpu-%E6%A8%A1%E5%BC%8F%E4%B8%8E%E6%9B%B4%E5%BF%AB%E7%9A%84-hami-core" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="enpu面向昇腾的策略化隔离">ENPU：面向昇腾的策略化隔离<a href="https://project-hami.io/zh/blog/hami-v2-11-0-release#enpu%E9%9D%A2%E5%90%91%E6%98%87%E8%85%BE%E7%9A%84%E7%AD%96%E7%95%A5%E5%8C%96%E9%9A%94%E7%A6%BB" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>v2.11.0 新增了第三种昇腾切分模式：<strong>ENPU</strong>（<a href="https://github.com/Project-HAMi/HAMi/pull/3110" target="_blank" rel="noopener noreferrer" class="">#3110</a>，<a href="https://github.com/maverick-woo" target="_blank" rel="noopener noreferrer" class="">@maverick-woo</a>，配套更新了 <a href="https://github.com/Project-HAMi/ascend-device-plugin" target="_blank" rel="noopener noreferrer" class="">ascend-device-plugin</a>）。带有 <code>huawei.com/vnpu-mode: enpu</code> 注解的 Pod 使用现有的昇腾数量、显存与算力资源，且只会被放置到支持 ENPU 的节点上：</p>
<ul>
<li class=""><strong>三种隔离策略</strong>：<code>fixed-share</code>、<code>elastic</code>、<code>best-effort</code>（通过 <code>huawei.com/enpu-policy</code> 注解或 chart 默认值 <code>devices.ascend.enpuPolicy</code> 选择），并在 admission 阶段持久化，后续修改默认值不会重新分类已有租户。</li>
<li class=""><strong>DIE 隔离</strong>：策略不兼容的 ENPU 租户之间、hami-core 与 ENPU 租户之间，会被隔离在不同的物理 DIE 上，包括没有模式注解的存量 hami-core Pod。A3 类硬件上保留单 DIE ENPU 请求，其他模式保持 SuperPod 行为。</li>
<li class=""><strong>不做模板取整</strong>：显存与算力配额直接透传，不做硬件模板取整；<code>devices.ascend.deviceSplitCount</code> 现在同时控制 <code>hami-core</code> 与 <code>enpu</code> 模式下每颗 NPU 的虚拟设备槽数（<a href="https://github.com/spencercjh" target="_blank" rel="noopener noreferrer" class="">@spencercjh</a>）。</li>
</ul>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">devices</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">ascend</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">enabled</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token boolean important" style="color:rgb(255, 88, 116)">true</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">hamiVnpuMode</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"enpu"</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># template | hami-core | enpu</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">enpuPolicy</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"elastic"</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># fixed-share | elastic | best-effort</span><br></div></code></pre></div></div>
<p>全局模式通过 <code>devices.ascend.hamiVnpuMode</code> 选择；来自 Ascend device plugin 的节点能力注解仍然优先。<code>devices.ascend.enabled</code> 只是在 HAMi 调度器中开启昇腾支持，并不会部署 device plugin：请单独安装 <a href="https://github.com/Project-HAMi/ascend-device-plugin" target="_blank" rel="noopener noreferrer" class="">ascend-device-plugin</a>，ENPU 模式需要 chart 0.2.0 及以上（0.2.x 同时包含 <code>devices.ascend.deviceSplitCount</code> 所依赖的 <code>vnpus.deviceSplitCount</code> 支持），并确保其镜像打包了与你的 CANN 版本匹配的 ENPU 运行时。前置条件请参阅<a class="" href="https://project-hami.io/zh/docs/next/userguide/ascend-device/enable-ascend-sharing#enpu">昇腾指南</a>。昇腾侧其他更新：chart 暴露 hami-core 模式的槽位容量；<strong>昇腾 950 支持</strong>也在本周期落地：调度器新增 Ascend950DT 整卡调度（<a href="https://github.com/Project-HAMi/HAMi/pull/3316" target="_blank" rel="noopener noreferrer" class="">#3316</a>，<a href="https://github.com/panpan0000" target="_blank" rel="noopener noreferrer" class="">@panpan0000</a>），<a href="https://github.com/Project-HAMi/ascend-device-plugin" target="_blank" rel="noopener noreferrer" class="">ascend-device-plugin</a> 0.2.x 增加 Ascend950DT 设备发现、节点模式覆盖，以及发现失败时保留不健康设备。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="hami-corelibvnpu性能优化">HAMi-core（libvNPU）性能优化<a href="https://project-hami.io/zh/blog/hami-v2-11-0-release#hami-corelibvnpu%E6%80%A7%E8%83%BD%E4%BC%98%E5%8C%96" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>在 LFX Mentorship 2026 Term 3 框架下，<a href="https://github.com/maverick-woo" target="_blank" rel="noopener noreferrer" class="">@maverick-woo</a> 重构了 <a href="https://github.com/Project-HAMi/hami-vnpu-core" target="_blank" rel="noopener noreferrer" class="">hami-vnpu-core</a> 的多个核心部分（<a href="https://github.com/xrwang8" target="_blank" rel="noopener noreferrer" class="">@xrwang8</a> 参与修复）：</p>
<ul>
<li class=""><strong>降低初始化锁竞争</strong>：fork 感知的每进程客户端、同流 worker 读锁快速路径、有界的调度轮次。</li>
<li class=""><strong>协作式空间计算策略与可补充透支策略</strong>，实现更细粒度的算力与显存控制。</li>
<li class=""><strong>按 NPU 的资源限制</strong>，同时覆盖缓存与运行时 HBM 分配路径，并计入 ACL <code>HostArgs</code> 的急切启动。</li>
</ul>
<p>这些改动面向 910C 多芯片切分场景，让软切分昇腾在并发推理下的性能显著提升。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="numa-refitcpu-与-gpu-调度终于对齐">NUMA Refit：CPU 与 GPU 调度终于对齐<a href="https://project-hami.io/zh/blog/hami-v2-11-0-release#numa-refitcpu-%E4%B8%8E-gpu-%E8%B0%83%E5%BA%A6%E7%BB%88%E4%BA%8E%E5%AF%B9%E9%BD%90" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>Kubernetes 的 CPU Manager 和 Topology Manager 会做出 NUMA 本地化的决策，但这些都发生在 HAMi 调度器<strong>已经选定 GPU 之后</strong>。当 kubelet 把分配限制到调度器未选择的 GPU 时，Pod 要么回退，要么带着跨 NUMA 的错位运行。v2.11.0 用 <strong>NUMA refit</strong> 打通了这个缺口（<a href="https://github.com/Project-HAMi/HAMi/pull/2731" target="_blank" rel="noopener noreferrer" class="">#2731</a>，<a href="https://github.com/saiyam1814" target="_blank" rel="noopener noreferrer" class="">@saiyam1814</a>，修复 <a href="https://github.com/Project-HAMi/HAMi/issues/2080" target="_blank" rel="noopener noreferrer" class="">#2080</a>，完成路线图中的 "Numa align (CPU and GPU)" 事项）：</p>
<p><img decoding="async" loading="lazy" alt="NUMA refit 流程" src="https://project-hami.io/zh/assets/images/numa-refit-flow-57d5732f25d3f43346a21d384eafb07e.svg#light-only" width="961" height="561" class="img_ev3q"> <img decoding="async" loading="lazy" alt="NUMA refit 流程" src="https://project-hami.io/zh/assets/images/numa-refit-flow-dark-7fc532dc753afb910bdf199eeee1cd95.svg#dark-only" width="961" height="561" class="img_ev3q"></p>
<p>调度器始终掌握主导权：它在 kubelet 允许的设备范围内重跑正常 fit，在一次 patch 中同时更新两份分配注解，并同步迁移内存中的预留，让运行时与账目保持一致。<code>nvidia.com/use-gpuuuid</code> 等设备固定会被尊重。该特性默认关闭，按 Pod 选择启用：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">metadata</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">annotations</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">hami.io/numa-alignment</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"strict"</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 或 best-effort</span><br></div></code></pre></div></div>
<p>同时需要在 chart 中设置 <code>devicePlugin.numaRefit.enabled=true</code>，并启用 NUMA 拓扑：使用默认设备配置时设置 <code>devices.nvidia.enableNumaTopology=true</code>；若 <code>device-config.content</code> 或 <code>files/device-config.yaml</code> 替换了生成的配置，则在自定义文件中设置 <code>nvidia.enableNumaTopology: true</code>（或在匹配的节点条目中设置 <code>enablenumatopology: true</code>）。节点配置中还需要 <code>operatingmode: hami-core</code> 与 <code>enablegetpreferredallocation</code>。NUMA refit 仅支持 <code>hami-core</code> 模式的节点，MIG 模式的节点不受支持。<code>strict</code> 在 refit 无法满足拓扑约束时让分配失败；<code>best-effort</code> 保持现有的回退行为。</p>
<p>该特性已在真实硬件上验证（8x NVIDIA RTX PRO 6000 Blackwell Server Edition、驱动 610.43.02、Kubernetes v1.35.6、CPU Manager <code>static</code>、Topology Manager <code>single-numa-node</code>）：带注解的 Pod 收到 <code>NumaRefitSucceed</code> 事件，最终 cpuset、运行时 GPU 与注解都落在同一个 NUMA 节点上。同周期内 <a href="https://github.com/asadjan4611" target="_blank" rel="noopener noreferrer" class="">@asadjan4611</a> 与 <a href="https://github.com/AyushSrivastava1818" target="_blank" rel="noopener noreferrer" class="">@AyushSrivastava1818</a> 还完成了一批加固（阶段感知的账目、配额校验、<code>/refit</code> 调用方认证），<a href="https://github.com/saiyam1814" target="_blank" rel="noopener noreferrer" class="">@saiyam1814</a> 修复了 Pod 删除时的 allocLock 问题。详见 <a class="" href="https://project-hami.io/zh/docs/userguide/nvidia-device/numa-alignment">NUMA 对齐指南</a>。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="flexible-mig-走向成熟">Flexible MIG 走向成熟<a href="https://project-hami.io/zh/blog/hami-v2-11-0-release#flexible-mig-%E8%B5%B0%E5%90%91%E6%88%90%E7%86%9F" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>Flexible MIG（动态 MIG 实例分配与释放）在 v2.10.0 发布时有已知限制：不支持 CDI、多设备场景未验证。v2.11.0 补齐了这些缺口：</p>
<ul>
<li class=""><strong>动态 MIG 设备支持 CDI</strong>（<a href="https://github.com/Project-HAMi/HAMi/pull/3070" target="_blank" rel="noopener noreferrer" class="">#3070</a>，<a href="https://github.com/asadjan4611" target="_blank" rel="noopener noreferrer" class="">@asadjan4611</a>）：HAMi 创建 MIG GI/CI 后，会发布原子化的每 MIG CDI 规范，并把 CDI 设备名返回给支持 CDI 的容器运行时。启动恢复会在 device plugin 服务分配前对账存活的 MIG 实例，分配失败会同时回滚 MIG 与 CDI 资源。v2.10.0 的 CDI 限制就此消除。</li>
<li class=""><strong>带回退的放置打分</strong>（<a href="https://github.com/Project-HAMi/HAMi/pull/2855" target="_blank" rel="noopener noreferrer" class="">#2855</a>，<a href="https://github.com/maishivamhoo123" target="_blank" rel="noopener noreferrer" class="">@maishivamhoo123</a>）：MIG 放置现在会打分，贪心选择不合适时回退重试；拓扑调度中重复的 MIG 候选会被去重。</li>
<li class=""><strong>按 Pod 的 profile 偏好</strong>（<a href="https://github.com/Project-HAMi/HAMi/pull/3014" target="_blank" rel="noopener noreferrer" class="">#3014</a>，<a href="https://github.com/maishivamhoo123" target="_blank" rel="noopener noreferrer" class="">@maishivamhoo123</a>）：Pod 可以通过 <code>nvidia.com/mig-profile-preference</code> 注解指定偏好的 MIG profile。调度器会优先尝试该 profile，无法分配时回退到正常的选择规则（显存请求、白名单中的 profile、放置约束）。</li>
<li class=""><strong>生命周期与监控</strong>：动态 MIG 实例生命周期得到优化（<a href="https://github.com/Project-HAMi/HAMi/pull/3128" target="_blank" rel="noopener noreferrer" class="">#3128</a>，<a href="https://github.com/asadjan4611" target="_blank" rel="noopener noreferrer" class="">@asadjan4611</a>），MIG 监控覆盖使用动态 MIG 的 Pod，vGPU 监控能正确发现 MIG 实例，分配会对照已注册的 profile 校验，白名单使用真实 NVML 名称（<a href="https://github.com/tsingst" target="_blank" rel="noopener noreferrer" class="">@tsingst</a>、<a href="https://github.com/FouoF" target="_blank" rel="noopener noreferrer" class="">@FouoF</a>、<a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">@archlitchi</a>、<a href="https://github.com/moezdil" target="_blank" rel="noopener noreferrer" class="">@moezdil</a>、<a href="https://github.com/usr-bin-ksh" target="_blank" rel="noopener noreferrer" class="">@usr-bin-ksh</a> 参与贡献）。</li>
<li class=""><strong>可靠性</strong>：GPU 出现 XID 错误时，其所有副本与 MIG 切片都会被标记为不健康；hami-core 节点上会自动禁用 MIG，避免生命周期控制冲突。</li>
</ul>
<p><a href="https://github.com/FouoF" target="_blank" rel="noopener noreferrer" class="">@FouoF</a> 还贡献了 <code>nvidiaDriverRoot</code> 自动检测（现为默认值，支持 GPU Operator driver-ready 契约）以及 NVML 会话配对修复。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="更安全的调度器">更安全的调度器<a href="https://project-hami.io/zh/blog/hami-v2-11-0-release#%E6%9B%B4%E5%AE%89%E5%85%A8%E7%9A%84%E8%B0%83%E5%BA%A6%E5%99%A8" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>v2.11.0 落地了安全审计驱动下的第一批调度器 extender 与 webhook 加固（主要由 <a href="https://github.com/moezdil" target="_blank" rel="noopener noreferrer" class="">@moezdil</a> 完成）：</p>
<ul>
<li class="">extender 的 HTTP 端点（<code>/filter</code> 与 <code>/bind</code>）只在<strong>本机回环地址上提供服务</strong>（<code>127.0.0.1:9444</code>），不再暴露到面向集群的端口。</li>
<li class="">extender 在处理请求前<strong>校验 Pod 身份</strong>。</li>
<li class="">带有<strong>伪造的调度器分配注解</strong>的 Pod 会在 admission 被拒绝，并在更新时受到保护。</li>
<li class="">节点锁路径只服务<strong>锁所指定的那个 Pod</strong>，过期锁的恢复被串行化，格式错误的设备分配注解会被拒绝。</li>
<li class="">chart 现在要求 <strong>Kubernetes 1.23+</strong>，守护 PSP，并对从第三方拉取的辅助镜像（certgen、Lupine 客户端与服务器镜像、mock device plugin）按 digest 固定；HAMi 自身镜像仍使用 <code>v2.11.0</code> 标签。</li>
</ul>
<p>调度器性能同样有提升：用量快照不再深拷贝整个 Pod（<a href="https://github.com/togettoyou" target="_blank" rel="noopener noreferrer" class="">@togettoyou</a>），快照只为候选节点构建，并在 CI 中加入基准回归跟踪（<a href="https://github.com/im-Toqeer-506" target="_blank" rel="noopener noreferrer" class="">@im-Toqeer-506</a>），节点打分尊重 preferred <code>nodeAffinity</code>（<a href="https://github.com/falconlee236" target="_blank" rel="noopener noreferrer" class="">@falconlee236</a>）。</p>
<p>v2.10.0 引入的调度策略，即 PodGroup 感知绑定、<code>mutex</code> 策略以及逗号分隔的策略组合，现在有了专门的调度策略 e2e 测试套件（<a href="https://github.com/Project-HAMi/HAMi/pull/2793" target="_blank" rel="noopener noreferrer" class="">#2793</a>，<a href="https://github.com/haitwang-cloud" target="_blank" rel="noopener noreferrer" class="">@haitwang-cloud</a>），并通过两处修复保证行为可预期：遇到无法识别的策略注解时保留已配置的策略（<a href="https://github.com/Project-HAMi/HAMi/pull/2769" target="_blank" rel="noopener noreferrer" class="">#2769</a>，<a href="https://github.com/IIITManjeet" target="_blank" rel="noopener noreferrer" class="">@IIITManjeet</a>），带填充的策略注解会解析为规范名称（<a href="https://github.com/Project-HAMi/HAMi/pull/2854" target="_blank" rel="noopener noreferrer" class="">#2854</a>）。</p>
<p>还有两项运维方面的补充（<a href="https://github.com/im-Toqeer-506" target="_blank" rel="noopener noreferrer" class="">@im-Toqeer-506</a>）：为开启 leader 选举的多副本调度器提供 <strong>PodDisruptionBudget</strong>（<a href="https://github.com/Project-HAMi/HAMi/pull/2773" target="_blank" rel="noopener noreferrer" class="">#2773</a>），避免节点排空时所有副本同时被驱逐；以及一个<strong>脱敏的诊断包收集器</strong>（<code>hack/hami-support-bundle.sh</code>，<a href="https://github.com/Project-HAMi/HAMi/pull/3149" target="_blank" rel="noopener noreferrer" class="">#3149</a>），为支持场景收集版本、工作负载、事件、HAMi ConfigMap 与限定长度的组件日志，且绝不读取 Secret。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="平台chart-与更多设备">平台、Chart 与更多设备<a href="https://project-hami.io/zh/blog/hami-v2-11-0-release#%E5%B9%B3%E5%8F%B0chart-%E4%B8%8E%E6%9B%B4%E5%A4%9A%E8%AE%BE%E5%A4%87" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class=""><strong>OpenShift 与 SELinux 支持</strong>（<a href="https://github.com/Project-HAMi/HAMi/pull/2208" target="_blank" rel="noopener noreferrer" class="">#2208</a>，<a href="https://github.com/spencercjh" target="_blank" rel="noopener noreferrer" class="">@spencercjh</a>）：<code>platform.openshift=true</code> 创建所需的 SCC，<code>selinux.enabled</code> 为 SELinux 强制集群重打主机目录标签。</li>
<li class=""><strong>Cluster Autoscaler 支持</strong>（<a href="https://github.com/Project-HAMi/HAMi/pull/2874" target="_blank" rel="noopener noreferrer" class="">#2874</a>，<a href="https://github.com/shkatara" target="_blank" rel="noopener noreferrer" class="">@shkatara</a>）：开启 <code>devicePlugin.reportNodeCapacity</code> 后，GPU 显存与算力资源会上报到节点 <code>capacity</code>/<code>allocatable</code>，让 Cluster Autoscaler 的扩容模拟、仪表盘和配额工具看到真实的 GPU 份额，而不是不透明的设备数量。</li>
<li class=""><strong>Chart 重构</strong>（<a href="https://github.com/spencercjh" target="_blank" rel="noopener noreferrer" class="">@spencercjh</a>）：所有厂商配置迁移到 <code>devices.VENDOR</code> 之下并完成 helm-docs 文档化，<code>global.imageSource</code> 在上游与国内镜像源之间切换，节点配置支持按标签选择器匹配（<a href="https://github.com/usr-bin-ksh" target="_blank" rel="noopener noreferrer" class="">@usr-bin-ksh</a>）。这对自定义 <code>values.yaml</code> 是<strong>破坏性变更</strong>。</li>
<li class=""><strong>NVIDIA device plugin</strong>：可选的<strong>宿主机 PID broker</strong>（<a href="https://github.com/Project-HAMi/HAMi/pull/2417" target="_blank" rel="noopener noreferrer" class="">#2417</a>，<a href="https://github.com/iemAnshuman" target="_blank" rel="noopener noreferrer" class="">@iemAnshuman</a>）通过 <code>SO_PEERCRED</code> 把每个进程的宿主机 PID 交给 HAMi-core，无需挂载宿主机 procfs；启动时发现的 IMEX channel 现在会真正传递到分配响应中（<a href="https://github.com/Project-HAMi/HAMi/pull/2892" target="_blank" rel="noopener noreferrer" class="">#2892</a>，<a href="https://github.com/Eshiv-Pandey" target="_blank" rel="noopener noreferrer" class="">@Eshiv-Pandey</a>）。</li>
<li class=""><strong>更多设备</strong>：AWS Inferentia1 支持全部四个 NeuronCore（<a href="https://github.com/asadjan4611" target="_blank" rel="noopener noreferrer" class="">@asadjan4611</a>），恢复寒武纪 MLU 的模型感知调度并把大于 1 的数量按整卡 MLU 计量（<a href="https://github.com/Project-HAMi/HAMi/pull/3218" target="_blank" rel="noopener noreferrer" class="">#3218</a>，<a href="https://github.com/B-White84" target="_blank" rel="noopener noreferrer" class="">@B-White84</a>），摩尔线程 S5000 每卡显存单位可配置（<a href="https://github.com/wjluo" target="_blank" rel="noopener noreferrer" class="">@wjluo</a>），海光 DCU 资源在 chart 与文档中更名为 <strong>HCU</strong>（<a href="https://github.com/kinglong08" target="_blank" rel="noopener noreferrer" class="">@kinglong08</a>），并更新了默认算力值，使 DTK 26.10+ 在未请求算力资源时像 NVIDIA 一样共享算力（<a href="https://github.com/Project-HAMi/HAMi/pull/3285" target="_blank" rel="noopener noreferrer" class="">#3285</a>，<a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">@archlitchi</a>）。其他厂商修复包括：沐曦（多容器 Pod 分散到独占 GPU，<a href="https://github.com/Project-HAMi/HAMi/pull/2837" target="_blank" rel="noopener noreferrer" class="">#2837</a>）、燧原（DRS profile 算力成本不超出设备预算，<a href="https://github.com/Project-HAMi/HAMi/pull/2810" target="_blank" rel="noopener noreferrer" class="">#2810</a>）、昆仑芯（拓扑选择不依赖切片顺序，<a href="https://github.com/Project-HAMi/HAMi/pull/2811" target="_blank" rel="noopener noreferrer" class="">#2811</a>、<a href="https://github.com/Project-HAMi/HAMi/pull/2924" target="_blank" rel="noopener noreferrer" class="">#2924</a>）、天数智芯（按每卡读取缩放后的算力上限，<a href="https://github.com/Project-HAMi/HAMi/pull/2961" target="_blank" rel="noopener noreferrer" class="">#2961</a>）。</li>
<li class=""><strong>监控</strong>：整卡 Pod 可以绕过 HAMi hook，通过直接 NVML 监控，覆盖带 <code>CUDA_DISABLE_CONTROL</code> 与动态 MIG 的 Pod（<a href="https://github.com/vesta-mlp" target="_blank" rel="noopener noreferrer" class="">@vesta-mlp</a>）；暴露硬件健康与错误指标（<a href="https://github.com/Gurkaran18" target="_blank" rel="noopener noreferrer" class="">@Gurkaran18</a>）；新增未知 GPU 进程计数指标（<a href="https://github.com/heuristicwave" target="_blank" rel="noopener noreferrer" class="">@heuristicwave</a>）；调度器暴露 leader 选举与缓存同步指标（<a href="https://github.com/KunwarSidhu47" target="_blank" rel="noopener noreferrer" class="">@KunwarSidhu47</a>）。</li>
<li class=""><strong>原生 sidecar 容器</strong>纳入 GPU 资源计算（<a href="https://github.com/maishivamhoo123" target="_blank" rel="noopener noreferrer" class="">@maishivamhoo123</a>），支持 Pod 级与容器级的环境变量覆盖（<a href="https://github.com/peachest" target="_blank" rel="noopener noreferrer" class="">@peachest</a>）。</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="project-hami-生态全景">Project-HAMi 生态全景<a href="https://project-hami.io/zh/blog/hami-v2-11-0-release#project-hami-%E7%94%9F%E6%80%81%E5%85%A8%E6%99%AF" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>本周期中，与 HAMi 配套发布的周边项目同样有不少进展：</p>
<ul>
<li class=""><strong><a href="https://github.com/Project-HAMi/HAMi-core" target="_blank" rel="noopener noreferrer" class="">HAMi-core</a></strong>（libvgpu.so）：SM 限流器改为按线程数而非 grid 数计费，CUDA graph 与 <code>cuMemAllocAsync</code> 的分配按 graph 生命周期跟踪与释放，VMM 映射在跟踪失败时干净回滚，<code>CUDA_DEVICE_MEMORY_LIMIT</code> 的后缀严格解析，修复了可能拿到其他进程 PID 的宿主机 PID 探测竞态，并提供与上文 device-plugin 特性配套的宿主机 PID broker 客户端。</li>
<li class=""><strong><a href="https://github.com/Project-HAMi/HAMi-WebUI" target="_blank" rel="noopener noreferrer" class="">HAMi-WebUI</a> v1.3.0</strong>：<code>/metrics</code> 抓取安全且减少页面侧查询扇出，Pod 设备注解可解码 init 容器，修正任务算力利用率，支持按 Pod 与容器名搜索，chart 在 ServiceMonitor 中保留 HAMi 工作负载标签并抓取内置的 kube-state-metrics。</li>
<li class=""><strong><a href="https://github.com/Project-HAMi/HAMi-dra" target="_blank" rel="noopener noreferrer" class="">HAMi-dra</a></strong>：chart 0.2.3 带来 webhook PodDisruptionBudget 和加固后的 <code>nouse-gpuuuid</code>/<code>nouse-gputype</code> CEL 选择器；此后主分支新增 AMD 支持（<code>drivers.amd</code>、<code>amd.com/gpu</code> claim 转换、<code>gpumem-percentage</code>）、<code>hami.io/skip-dra</code> 退出开关，以及与 HAMi 和 Prometheus 约定对齐的监控指标标签。</li>
<li class=""><strong><a href="https://github.com/Project-HAMi/k8s-dra-driver" target="_blank" rel="noopener noreferrer" class="">k8s-dra-driver</a> v0.1.1</strong>：为 DRA 路径提供 vGPU 监控、清理遗留 claim 目录、修复缺失 PCIe root 属性的问题，并在 kind + Mokka 上增加无 GPU 的 e2e 测试。<strong><a href="https://github.com/Project-HAMi/ascend-dra-driver" target="_blank" rel="noopener noreferrer" class="">ascend-dra-driver</a></strong> chart 0.2.0 通过与 claim Prepare/Unprepare 对齐的内部收集器暴露 <code>/metrics</code>，为 HAMi vNPU-core 模式增加 DeviceClass，并在 ResourceSlice 中上报真实 UUID。</li>
<li class=""><strong><a href="https://github.com/Project-HAMi/volcano-vgpu-device-plugin" target="_blank" rel="noopener noreferrer" class="">volcano-vgpu-device-plugin</a> v1.13.0</strong>：kubelet 流重连时保留健康状态，校验分配状态与设备注解，<code>WatchAndRegister</code> 随插件干净停止，并更新到最新的 HAMi-core。</li>
<li class=""><strong><a href="https://github.com/Project-HAMi/gpu-descheduler" target="_blank" rel="noopener noreferrer" class="">gpu-descheduler</a> v0.0.2</strong>：新项目，通过 mutating webhook 与 Placement CRD 把分散在各节点上的 vGPU 工作负载整理归拢；本周期增加多架构构建，并改用 HAMi 的 AMD <code>use-gpuuuid</code> 注解键。<strong><a href="https://github.com/Project-HAMi/kai-resource-isolator" target="_blank" rel="noopener noreferrer" class="">kai-resource-isolator</a></strong> 为多副本部署增加 webhook PodDisruptionBudget 与反亲和，并校验 chart 输入。</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="重构与重绘后的官网">重构与重绘后的官网<a href="https://project-hami.io/zh/blog/hami-v2-11-0-release#%E9%87%8D%E6%9E%84%E4%B8%8E%E9%87%8D%E7%BB%98%E5%90%8E%E7%9A%84%E5%AE%98%E7%BD%91" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>与代码发布同步，<a href="https://project-hami.io/" target="_blank" rel="noopener noreferrer" class="">project-hami.io</a> 完成了上线以来最大的一次改版：</p>
<ul>
<li class=""><strong>全新信息架构</strong>（<a href="https://github.com/moezdil" target="_blank" rel="noopener noreferrer" class="">@moezdil</a>）：文档侧边栏重组为引导式学习路径，设备矩阵成为唯一的厂商索引（取代按分类罗列），每个厂商指南拥有独立侧边栏。HAMi 仓库内的全部文档统一迁移到官网，内部设计文档移植为公开的开发者章节，并补充了中文翻译。四个过时的内部页面随之移除。</li>
<li class=""><strong>重绘的 Concepts 章节</strong>（<a href="https://github.com/togettoyou" target="_blank" rel="noopener noreferrer" class="">@togettoyou</a>）：旧的 Introduction 页面被 Concepts 分类取代，讲解 device plugin 与 DRA 背景，围绕一个 Pod 的完整旅程重写架构页，覆盖设备共享资源模型与 HAMi-core 的显存、算力隔离原理，全部配以全新绘制的 SVG 图示。配套的图示品牌规范（含暗色模式 token 与 Mermaid 样式）保证新图示视觉一致。</li>
<li class=""><strong>面向 LLM 的 <code>llms.txt</code></strong>（<a href="https://github.com/spencercjh" target="_blank" rel="noopener noreferrer" class="">@spencercjh</a>）：官网现在发布 <code>llms.txt</code>，让编码代理与 LLM 工具可以高效消费整套文档。</li>
<li class=""><strong>重构的安装与升级指南</strong>：专门的<a class="" href="https://project-hami.io/zh/docs/next/installation/upgrade">升级指南</a>重建为带破坏性变更提示的分步流程，离线安装说明重构（<a href="https://github.com/spencercjh" target="_blank" rel="noopener noreferrer" class="">@spencercjh</a>），用户指南配置页重组，NVIDIA 指南新增功能支持矩阵。</li>
<li class=""><strong>平台安装指南</strong>：<a href="https://github.com/spencercjh" target="_blank" rel="noopener noreferrer" class="">@spencercjh</a> 贡献了 OpenShift、GKE、K3s 指南；<a href="https://github.com/togettoyou" target="_blank" rel="noopener noreferrer" class="">@togettoyou</a> 贡献了 TKE、ACK、CCE 等国产云指南与 HAMi 实操 Workshop。</li>
<li class=""><strong>Grafana 仪表盘整合</strong>（<a href="https://github.com/togettoyou" target="_blank" rel="noopener noreferrer" class="">@togettoyou</a>），新增 MIG 库存与按 profile 使用面板、资源配额用量对比表和 vGPU 算力面板（<a href="https://github.com/loiht2" target="_blank" rel="noopener noreferrer" class="">@loiht2</a>）。</li>
<li class=""><strong>官网新内容</strong>：赛力斯（Seres）案例、CETC Cloud 加入贡献者名单、ZYT 加入采用者名单，以及社区大使计划。</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="v210-以来的社区活动">v2.10 以来的社区活动<a href="https://project-hami.io/zh/blog/hami-v2-11-0-release#v210-%E4%BB%A5%E6%9D%A5%E7%9A%84%E7%A4%BE%E5%8C%BA%E6%B4%BB%E5%8A%A8" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>从 v2.10.0 到 v2.11.0 的这段时间，是项目历史上最忙碌的时期之一：</p>
<ul>
<li class=""><strong><a href="https://www.lfopensource.cn/kubecon-cloudnativecon-openinfra-summit-pytorch-conference-china/" target="_blank" rel="noopener noreferrer" class="">KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026</a></strong>（9 月，上海）：HAMi 成为 CNCF 孵化项目后首次参加 KubeCon China，带来两个 keynote、一个 lightning talk、一场万卡生产环境分享，招商银行案例在 keynote 舞台上获得案例大奖。阅读<a class="" href="https://project-hami.io/zh/blog/kubecon-china-2026-recap">回顾</a>。</li>
<li class=""><strong>LFX Mentorship 2026 Term 3</strong>（9 月至 11 月）：四个学员项目，覆盖 HAMi-core 锁竞争、GPU 显存隔离、可观测性与文档。本周期的成果已经进入 v2.11.0，包括前文的 HAMi-core 性能优化。见<a class="" href="https://project-hami.io/zh/blog/lfx-mentorship-2026-term-3">项目介绍</a>。</li>
<li class=""><strong>ContainerDays &amp; AI Context Singapore 2026</strong>（10 月 27-28 日）：HAMi 出现在五个环节中，包括 <a href="https://github.com/rootsongjc" target="_blank" rel="noopener noreferrer" class="">@rootsongjc</a> 与 <a href="https://github.com/fishman" target="_blank" rel="noopener noreferrer" class="">@fishman</a> 带来的 GPU 调度动手 workshop。阅读<a class="" href="https://project-hami.io/zh/blog/containerdays-singapore-2026-recap">回顾</a>。</li>
<li class=""><strong>DRA NPU 共享</strong>：社区发布了 <a class="" href="https://project-hami.io/zh/blog/hami-dra-npu-sharing">HAMi-DRA NPU 共享</a>深度文章，把 DRA 故事延伸到昇腾 NPU。</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="贡献者">贡献者<a href="https://project-hami.io/zh/blog/hami-v2-11-0-release#%E8%B4%A1%E7%8C%AE%E8%80%85" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>v2.11.0 是 HAMi 主仓库与 Project-HAMi 组织（ascend-device-plugin、hami-vnpu-core、HAMi-core、HAMi-dra、amd-device-plugin、amd-dra-driver、HAMi-WebUI 等）共同贡献的成果。本周期代表性功能负责人包括：</p>
<ul>
<li class=""><a href="https://github.com/moezdil" target="_blank" rel="noopener noreferrer" class="">@moezdil</a>：Remote GPU 调度器后端与 device-plugin 集成、AMD 特性矩阵、调度器安全加固、发布工程，以及官网信息架构重组</li>
<li class=""><a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">@archlitchi</a>：发布协调、vNPU 模板、remote 客户端工具、监控修复</li>
<li class=""><a href="https://github.com/maverick-woo" target="_blank" rel="noopener noreferrer" class="">@maverick-woo</a>：昇腾 ENPU 调度、hami-vnpu-core 性能优化（空间计算策略、按 NPU 限制、锁竞争优化）</li>
<li class=""><a href="https://github.com/saiyam1814" target="_blank" rel="noopener noreferrer" class="">@saiyam1814</a>：NUMA refit（CPU 与 GPU 调度对齐）</li>
<li class=""><a href="https://github.com/FouoF" target="_blank" rel="noopener noreferrer" class="">@FouoF</a>：Flexible MIG 增强、remote 模式增强、驱动根目录自动检测</li>
<li class=""><a href="https://github.com/asadjan4611" target="_blank" rel="noopener noreferrer" class="">@asadjan4611</a>：动态 MIG 的 CDI 支持、Lupine 资源池、Inferentia 与寒武纪修复、NUMA refit 加固</li>
<li class=""><a href="https://github.com/spencercjh" target="_blank" rel="noopener noreferrer" class="">@spencercjh</a>：OpenShift 与 SELinux 支持、重构为 <code>devices.VENDOR</code> 的 chart 与全量 helm-docs、hami-core/ENPU 的 <code>deviceSplitCount</code>、<code>llms.txt</code>、离线安装与用户指南文档重构、OpenShift/GKE/K3s 安装指南、DCU 更名为 HCU 的文档迁移</li>
<li class=""><a href="https://github.com/maishivamhoo123" target="_blank" rel="noopener noreferrer" class="">@maishivamhoo123</a>：MIG 放置打分与 profile 偏好、原生 sidecar 资源计量</li>
<li class=""><a href="https://github.com/sanket-jadhav-cse" target="_blank" rel="noopener noreferrer" class="">@sanket-jadhav-cse</a>：多容器 Lupine 分配修复</li>
<li class=""><a href="https://github.com/shkatara" target="_blank" rel="noopener noreferrer" class="">@shkatara</a>：节点容量资源上报</li>
<li class=""><a href="https://github.com/im-Toqeer-506" target="_blank" rel="noopener noreferrer" class="">@im-Toqeer-506</a>：调度器性能与基准测试、PodDisruptionBudget、诊断包</li>
<li class=""><a href="https://github.com/togettoyou" target="_blank" rel="noopener noreferrer" class="">@togettoyou</a>：HAMi 侧的调度器深拷贝消除与 chart 文档；官网侧的重绘 Concepts 章节、Grafana 仪表盘整合、HAMi Workshop 教程，以及 TKE/ACK/CCE 安装指南</li>
<li class=""><a href="https://github.com/vesta-mlp" target="_blank" rel="noopener noreferrer" class="">@vesta-mlp</a>：整卡监控绕过 hook</li>
<li class=""><a href="https://github.com/xrwang8" target="_blank" rel="noopener noreferrer" class="">@xrwang8</a>：hami-vnpu-core 配额与调度修复</li>
<li class=""><a href="https://github.com/panpan0000" target="_blank" rel="noopener noreferrer" class="">@panpan0000</a>：Ascend950DT 整卡调度</li>
<li class=""><a href="https://github.com/kinglong08" target="_blank" rel="noopener noreferrer" class="">@kinglong08</a>：海光 DCU 更名为 HCU</li>
<li class=""><a href="https://github.com/wjluo" target="_blank" rel="noopener noreferrer" class="">@wjluo</a>：摩尔线程 S5000 支持</li>
<li class=""><a href="https://github.com/usr-bin-ksh" target="_blank" rel="noopener noreferrer" class="">@usr-bin-ksh</a>：nodeconfig 标签选择器、MIG 白名单</li>
</ul>
<p>本版本有 29 位贡献者完成首次贡献，包括 <a href="https://github.com/Kevindua26" target="_blank" rel="noopener noreferrer" class="">@Kevindua26</a>、<a href="https://github.com/im-Toqeer-506" target="_blank" rel="noopener noreferrer" class="">@im-Toqeer-506</a>、<a href="https://github.com/togettoyou" target="_blank" rel="noopener noreferrer" class="">@togettoyou</a>、<a href="https://github.com/Woolgathererer" target="_blank" rel="noopener noreferrer" class="">@Woolgathererer</a>、<a href="https://github.com/curry30000" target="_blank" rel="noopener noreferrer" class="">@curry30000</a>、<a href="https://github.com/sanket-jadhav-cse" target="_blank" rel="noopener noreferrer" class="">@sanket-jadhav-cse</a>、<a href="https://github.com/usr-bin-ksh" target="_blank" rel="noopener noreferrer" class="">@usr-bin-ksh</a>、<a href="https://github.com/vesta-mlp" target="_blank" rel="noopener noreferrer" class="">@vesta-mlp</a>、<a href="https://github.com/tsingst" target="_blank" rel="noopener noreferrer" class="">@tsingst</a>、<a href="https://github.com/loiht2" target="_blank" rel="noopener noreferrer" class="">@loiht2</a>、<a href="https://github.com/falconlee236" target="_blank" rel="noopener noreferrer" class="">@falconlee236</a>、<a href="https://github.com/heuristicwave" target="_blank" rel="noopener noreferrer" class="">@heuristicwave</a>。感谢每一位贡献者，也感谢每一位报告问题、测试候选版本的用户。完整名单见 <a href="https://github.com/Project-HAMi/HAMi/releases/tag/v2.11.0" target="_blank" rel="noopener noreferrer" class="">v2.11.0 Release Notes</a>。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="升级指南">升级指南<a href="https://project-hami.io/zh/blog/hami-v2-11-0-release#%E5%8D%87%E7%BA%A7%E6%8C%87%E5%8D%97" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>通过 Helm 升级至 v2.11.0：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm repo </span><span class="token function" style="color:rgb(130, 170, 255)">add</span><span class="token plain"> hami-charts https://project-hami.github.io/HAMi/</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm repo update</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm upgrade hami hami-charts/hami </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> kube-system </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--version</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">2.11</span><span class="token plain">.0</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 如果你的 values.yaml 自定义过厂商配置，请先迁移到新的 devices.VENDOR 结构，</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 再显式传入。--reset-values 会丢弃 release 中保存的旧值，</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 因此所有需要保留的覆盖项都必须写进 my-values.yaml。</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm upgrade hami hami-charts/hami </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> kube-system </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--version</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">2.11</span><span class="token plain">.0 </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  --reset-values </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--values</span><span class="token plain"> my-values.yaml</span><br></div></code></pre></div></div>
<p>带破坏性变更提示的分步升级流程，请参阅专门的<a class="" href="https://project-hami.io/zh/docs/next/installation/upgrade">升级指南</a>。完整安装文档请参考：<a class="" href="https://project-hami.io/zh/docs/installation/online-installation">https://project-hami.io/zh/docs/installation/online-installation</a></p>
<div class="theme-admonition theme-admonition-warning admonition_xJq3 alert alert--warning"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 16 16"><path fill-rule="evenodd" d="M8.893 1.5c-.183-.31-.52-.5-.887-.5s-.703.19-.886.5L.138 13.499a.98.98 0 0 0 0 1.001c.193.31.53.501.886.501h13.964c.367 0 .704-.19.877-.5a1.03 1.03 0 0 0 .01-1.002L8.893 1.5zm.133 11.497H6.987v-2.003h2.039v2.003zm0-3.004H6.987V5.987h2.039v4.006z"></path></svg></span>升级注意事项</div><div class="admonitionContent_BuS1"><ul>
<li class=""><strong>Chart 破坏性变更</strong>：厂商配置迁移到了 <code>devices.VENDOR</code> 之下。如果你的 <code>values.yaml</code> 自定义过厂商设备配置，请在升级前完成迁移；chart 现在一旦发现旧的（已迁移的）字段路径就会渲染失败。</li>
<li class="">chart 要求 <strong>Kubernetes 1.23+</strong>。</li>
<li class=""><strong>MIG + hami-core</strong>：hami-core 节点上会自动禁用 MIG；若同时依赖两者，请核实节点配置。</li>
<li class=""><strong>海光用户</strong>：DCU 资源在 chart 与文档中更名为 HCU。</li>
<li class=""><strong>指标弃用</strong>：<code>hami_container_device_memory_bytes</code> 指标以及 <code>hami_resource_quota_used</code> 的 <code>limit</code> 标签已被弃用，请更新依赖它们的看板与告警。</li>
<li class=""><strong>指标标签变更</strong>：四个 vGPU 分配指标中恒为 0 的 <code>container_index</code>/<code>containeridx</code> 标签已移除（<a href="https://github.com/Project-HAMi/HAMi/pull/2898" target="_blank" rel="noopener noreferrer" class="">#2898</a>）；<code>hami_host_gpu_*</code> 的 <code>device_type</code> 标签不再带重复的 <code>NVIDIA</code> 前缀，从而与调度器的 <code>hami_gpu_*</code> 序列一致（<a href="https://github.com/Project-HAMi/HAMi/pull/2991" target="_blank" rel="noopener noreferrer" class="">#2991</a>）。请调整按这些标签分组或过滤的 PromQL。</li>
<li class=""><strong>更严格的请求校验</strong>：无效的 HAMi 设备请求现在会直接失败，而不是把无 GPU 的 Pod 绑定到节点（<a href="https://github.com/Project-HAMi/HAMi/pull/2994" target="_blank" rel="noopener noreferrer" class="">#2994</a>）；超出 0-100 的 GPU 算力百分比（<a href="https://github.com/Project-HAMi/HAMi/pull/2784" target="_blank" rel="noopener noreferrer" class="">#2784</a>）和带小数的 <code>nvidia.com/gpumem-percentage</code>（<a href="https://github.com/Project-HAMi/HAMi/pull/2789" target="_blank" rel="noopener noreferrer" class="">#2789</a>）会在准入阶段被拒绝；device plugin 会拒绝超过容器请求量的分配（<a href="https://github.com/Project-HAMi/HAMi/pull/3075" target="_blank" rel="noopener noreferrer" class="">#3075</a>）。此前因请求被静默丢弃而"正常运行"的工作负载现在会被拒绝。</li>
<li class=""><strong>严格的 device-config 解析</strong>：<code>device-config.yaml</code> 中未知或拼错的字段现在会让调度器启动失败并给出明确错误，而不是被忽略（<a href="https://github.com/Project-HAMi/HAMi/pull/2941" target="_blank" rel="noopener noreferrer" class="">#2941</a>）。升级前请检查自定义的 <code>device-config.content</code> 覆盖。</li>
<li class=""><strong>移除节点握手注解</strong>：不再写入 <code>hami.io/node-handshake-*</code> 注解，并在注册时清理；设备健康状态现在跟随节点的 <code>allocatable</code>（<a href="https://github.com/Project-HAMi/HAMi/pull/3227" target="_blank" rel="noopener noreferrer" class="">#3227</a>，AMD：<a href="https://github.com/Project-HAMi/HAMi/pull/3182" target="_blank" rel="noopener noreferrer" class="">#3182</a>）。请移除依赖这些注解的工具。</li>
<li class=""><strong>限制下发方式与缓存文件</strong>：<code>devices.nvidia.limitDelivery</code> 可选 <code>file</code>（默认，行为不变）或 <code>env</code>（仅环境变量，不再生成 <code>/overrideEnv</code> 文件），可按节点（nodeconfig 中的 <code>limitdelivery</code>）和按 Pod（<code>hami.io/limit-delivery</code>）覆盖；共享缓存文件现在固定命名为 <code>usage.cache</code>，不再使用随机名称（<a href="https://github.com/Project-HAMi/HAMi/pull/3226" target="_blank" rel="noopener noreferrer" class="">#3226</a>）。与其他 <code>devices.nvidia</code> 值一样，<code>limitDelivery</code> 会写入生成的设备配置；若 <code>device-config.content</code> 或 <code>files/device-config.yaml</code> 替换了它，请改在那里设置该字段。</li>
<li class=""><strong>ResourceQuota 语义</strong>：同一命名空间存在多个 ResourceQuota 时，生效的设备限额取其中的最小值，删除其中一个也不再清除限制（<a href="https://github.com/Project-HAMi/HAMi/pull/3094" target="_blank" rel="noopener noreferrer" class="">#3094</a>）；配额显存限额按 <code>memoryFactor</code> 缩放后存储，因此 <code>hami_resource_quota_limit</code> 现在与 <code>hami_resource_quota_used</code> 一致（<a href="https://github.com/Project-HAMi/HAMi/pull/3339" target="_blank" rel="noopener noreferrer" class="">#3339</a>）。</li>
<li class=""><strong>新增 chart 配置项</strong>：用于调优 kube-apiserver 客户端的 <code>scheduler.kubeQPS</code> / <code>kubeBurst</code> / <code>kubeTimeout</code>（<a href="https://github.com/Project-HAMi/HAMi/pull/2905" target="_blank" rel="noopener noreferrer" class="">#2905</a>）、<code>scheduler.nodeLockRetryTimeout</code> 与 <code>scheduler.extenderHTTPTimeout</code>（<a href="https://github.com/Project-HAMi/HAMi/pull/2944" target="_blank" rel="noopener noreferrer" class="">#2944</a>），以及面向私有仓库中 Remote GPU 镜像的拉取凭据：会话中继 Pod 使用 <code>devices.remotegpu.sessionImagePullSecrets</code>，Lupine 服务器 Pod 使用 <code>devices.remotegpu.server.image.pullSecrets</code>（<a href="https://github.com/Project-HAMi/HAMi/pull/3329" target="_blank" rel="noopener noreferrer" class="">#3329</a>）。调度器相关的值不受 <code>device-config.content</code> 覆盖的影响。</li>
<li class="">Remote GPU、ENPU 与 NUMA refit <strong>默认全部关闭</strong>，需显式启用。</li>
<li class="">建议在升级前于测试环境验证兼容性。</li>
</ul></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="总结">总结<a href="https://project-hami.io/zh/blog/hami-v2-11-0-release#%E6%80%BB%E7%BB%93" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>HAMi v2.11.0 是一次关于<strong>消除边界</strong>的版本：本地与远程 GPU 之间的边界（基于 Lupine 的 Remote GPU）、GPU 厂商之间的边界（与 NVIDIA 并列的 AMD Instinct 共享）、CPU 与 GPU 调度之间的边界（NUMA refit）、昇腾上不同切分方式之间的边界（支持策略隔离的 ENPU），以及动态 MIG 与现代容器运行时之间的边界（CDI 支持）。再加上安全加固的调度器、OpenShift/SELinux 支持、重构后的 chart，以及重组重绘后的官网，HAMi 持续拓展统一异构算力调度平台的能力边界。</p>
<p>我们诚挚欢迎更多开发者、用户和生态伙伴参与 HAMi 社区，共同推动 GPU 虚拟化与异构算力调度能力的演进。</p>
<hr>
<p><strong>相关链接：</strong></p>
<ul>
<li class="">GitHub Release：<a href="https://github.com/Project-HAMi/HAMi/releases/tag/v2.11.0" target="_blank" rel="noopener noreferrer" class="">https://github.com/Project-HAMi/HAMi/releases/tag/v2.11.0</a></li>
<li class="">Remote GPU 示例：<a href="https://github.com/Project-HAMi/HAMi/tree/master/examples/remotegpu" target="_blank" rel="noopener noreferrer" class="">https://github.com/Project-HAMi/HAMi/tree/master/examples/remotegpu</a></li>
<li class="">Lupine：<a href="https://github.com/lupinemachines" target="_blank" rel="noopener noreferrer" class="">https://github.com/lupinemachines</a></li>
<li class="">hami-vnpu-core：<a href="https://github.com/Project-HAMi/hami-vnpu-core" target="_blank" rel="noopener noreferrer" class="">https://github.com/Project-HAMi/hami-vnpu-core</a></li>
<li class="">项目文档：<a class="" href="https://project-hami.io/zh/docs">https://project-hami.io/zh/docs/</a></li>
<li class="">社区 Discord（推荐）：<a href="https://discord.gg/Amhy7XmbNq" target="_blank" rel="noopener noreferrer" class="">https://discord.gg/Amhy7XmbNq</a></li>
<li class="">社区 CNCF Slack：<a href="https://cloud-native.slack.com/archives/C07T10BU4R2" target="_blank" rel="noopener noreferrer" class="">https://cloud-native.slack.com/archives/C07T10BU4R2</a></li>
</ul>]]></content>
        <author>
            <name>HAMi 社区</name>
        </author>
        <category label="Release" term="Release"/>
        <category label="GPU" term="GPU"/>
        <category label="Kubernetes" term="Kubernetes"/>
        <category label="调度" term="调度"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[HAMi 亮相 ContainerDays 新加坡 2026：从 GPU 共享到 AI Factory]]></title>
        <id>https://project-hami.io/zh/blog/containerdays-singapore-2026-recap</id>
        <link href="https://project-hami.io/zh/blog/containerdays-singapore-2026-recap"/>
        <updated>2026-10-30T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[回顾 ContainerDays 新加坡 2026 上与 HAMi 相关的五场分享：GPU 共享动手实验、Neocloud 算力利用率、DRA 集成、AI Factory 架构与 AI 原生基础设施。]]></summary>
        <content type="html"><![CDATA[<p><a href="https://www.containerdays.io/containerdays-singapore-2026/" target="_blank" rel="noopener noreferrer" class="">ContainerDays &amp; AI Context Singapore 2026</a> 于 10 月 27 至 28 日在新加坡新达城会展中心举行，议题覆盖云原生、AI 基础设施与平台工程。HAMi Community Manager <a href="https://github.com/rootsongjc" target="_blank" rel="noopener noreferrer" class="">宋净超（Jimmy Song）</a>与 HAMi Contributor <a href="https://github.com/fishman" target="_blank" rel="noopener noreferrer" class="">Reza Jelveh</a>（两位均来自密瓜智能）带来了一场 Workshop 和一场技术分享，HAMi 还出现在大会的另外三场议程中。</p>
<p><img decoding="async" loading="lazy" alt="ContainerDays 新加坡 2026" src="https://project-hami.io/zh/assets/images/banner-1dba3e75d4e64d8ed595fd54ee320339.png" width="1280" height="715" class="img_ev3q"></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="workshop从零开始实践共享-gpu">Workshop：从零开始实践共享 GPU<a href="https://project-hami.io/zh/blog/containerdays-singapore-2026-recap#workshop%E4%BB%8E%E9%9B%B6%E5%BC%80%E5%A7%8B%E5%AE%9E%E8%B7%B5%E5%85%B1%E4%BA%AB-gpu" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>Workshop <strong>From Zero to Shared GPUs: Hands-on GPU Scheduling in Kubernetes with HAMi</strong> 通过基于 kind 的本地实验环境带领参与者实践 GPU 共享。实验使用模拟设备插件注册了两张虚拟的 NVIDIA A100-40GB，参与者无需物理 GPU，即可运行真实的 HAMi Scheduler，观察资源申请、设备选择与 Pod 调度的完整过程。</p>
<p><img decoding="async" loading="lazy" alt="ContainerDays 新加坡 2026 的 Workshop 现场" src="https://project-hami.io/zh/assets/images/workshop-demo-3ccbf0393341908a82d4d1d645307094.jpg" width="3424" height="2567" class="img_ev3q"></p>
<p>一组实验直观展示了调度中的关键问题：三个各申请 15,000 MB 显存与 30% 计算资源的 Pod 被分配到两张模拟 GPU 上，而一个申请 45,000 MB 显存的 Pod 始终处于 Pending 状态。即使两张卡的剩余显存之和能够满足请求，调度器也必须判断单个设备能否满足任务需求，而不仅是检查节点的空闲容量。</p>
<p>实验随后对比了 <strong>binpack</strong> 与 <strong>spread</strong> 两种放置策略的效果，并介绍了 HAMi 与 <strong>KAI Scheduler</strong>、<strong>Volcano</strong> 以及 <strong>Kubernetes DRA</strong> 的协作方式：无论选择哪种调度体系，都可以由 HAMi-core 提供运行时资源限制。模拟 GPU 适合验证调度逻辑，运行时隔离仍需在真实硬件上验证。</p>
<p><a href="https://project-hami.io/zh/assets/files/containerdays-singapore-2026-workshop-6a57bef641783dea45ed779b18b709eb.pdf" target="_blank" class="">工作坊讲义</a>与<a href="https://github.com/Project-HAMi/hami-workshop" target="_blank" rel="noopener noreferrer" class="">实验手册</a>已在线发布，详情见<a class="" href="https://project-hami.io/zh/landing/containerdays-singapore">活动专页</a>。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="分享让-neocloud-的-gpu-利用率提升十倍">分享：让 Neocloud 的 GPU 利用率提升十倍<a href="https://project-hami.io/zh/blog/containerdays-singapore-2026-recap#%E5%88%86%E4%BA%AB%E8%AE%A9-neocloud-%E7%9A%84-gpu-%E5%88%A9%E7%94%A8%E7%8E%87%E6%8F%90%E5%8D%87%E5%8D%81%E5%80%8D" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>Reza Jelveh 的分享 <strong>10X Your Neocloud's GPU Utilization with HAMi</strong> 关注 Neocloud 平台的实际问题：当 GPU 需要同时服务多个租户和不同类型的工作负载时，如何减少资源闲置？</p>
<p>演讲引用的多组数据显示，不同团队的 GPU 利用率大多在 5% 至 20% 之间：一部分任务还在排队等待 GPU，已分配出去的 GPU 却没有被充分使用。一个只需约 1 GB 显存的时序推理任务，在整卡分配模式下仍可能独占一张 141 GB 的 H200。</p>
<p><img decoding="async" loading="lazy" alt="推理副本共享同一张 GPU，利用率随副本数上升而 p95 延迟保持稳定" src="https://project-hami.io/zh/assets/images/inference-replicas-9480176ebebf570c6f33e199199bf3ba.png" width="2400" height="1350" class="img_ev3q"></p>
<p>对于推理服务，在同一张卡上增加副本可以提升利用率：演示中，10 个副本的 GPU SM 利用率达到 100%，p95 延迟保持不变。标题中的“10X”并非对所有平台的承诺，实际效果取决于工作负载构成、资源配置与平台基线。大型训练任务通常保留整卡，不可信租户需要结合 MIG 或独占设备评估隔离要求；面对 AMD、昇腾等异构设备，HAMi 提供对应的资源管理与限制机制，并可与 Volcano、Kueue、KAI Scheduler 等协同。演讲还区分了 HAMi 的两类监控指标：Scheduler 暴露的资源分配指标（端口 9395）与 vGPUmonitor 暴露的资源使用指标（端口 9394）。</p>
<p><a href="https://project-hami.io/zh/assets/files/containerdays-singapore-2026-10x-6c46aea08a600846825ebba333d4a99a.pdf" target="_blank" class="">演讲幻灯片</a>已在线发布，详情见<a class="" href="https://project-hami.io/zh/landing/containerdays-singapore-10x">活动专页</a>。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="dra-负责分配谁来负责执行">DRA 负责分配，谁来负责执行？<a href="https://project-hami.io/zh/blog/containerdays-singapore-2026-recap#dra-%E8%B4%9F%E8%B4%A3%E5%88%86%E9%85%8D%E8%B0%81%E6%9D%A5%E8%B4%9F%E8%B4%A3%E6%89%A7%E8%A1%8C" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>在 <strong>DRA Allocates, But Who Enforces? Closing the GPU Sharing Gap with HAMi</strong> 这场分享中，Ivan Porta（Todea）讨论了 DRA 的资源分配与运行时限制之间的衔接问题。DRA 可以通过 ResourceClaims 描述和分配设备，但 Kubernetes API 本身并不会阻止容器在运行时超出 GPU 资源额度。</p>
<p>演讲介绍了 HAMi 各组件如何补足这一环节：Mutating Webhook 调整调度相关信息，Scheduler Extender 参与调度决策，Device Plugin 负责设备注册与分配，HAMi-core 在容器运行时拦截 CUDA 调用并执行显存与计算资源限制。HAMi-DRA 还可以将 <code>nvidia.com/gpu</code> 资源请求转换为 DRA ResourceClaims，在保持 Pod 资源接口稳定的同时，让底层资源管理方式随 Kubernetes 环境演进。</p>
<p><img decoding="async" loading="lazy" alt="Ivan Porta 演讲现场" src="https://project-hami.io/zh/assets/images/ivan-porta-eddcd8e6d25e6fe9dd3eee768a39cba5.jpg" width="1440" height="1080" class="img_ev3q"></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="人人都会构建-ai-factory-吗">人人都会构建 AI Factory 吗？<a href="https://project-hami.io/zh/blog/containerdays-singapore-2026-recap#%E4%BA%BA%E4%BA%BA%E9%83%BD%E4%BC%9A%E6%9E%84%E5%BB%BA-ai-factory-%E5%90%97" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>Saiyam Pathak（vCluster Labs）分享了从个人电脑上的单张 GPU 到企业级 AI Factory 的演进路径。现场演示让多个团队各自隔离的 Kubernetes 集群与推理工作负载运行在同一张 GPU 上，其中使用了 vCluster 与 HAMi 等开源项目。</p>
<p><img decoding="async" loading="lazy" alt="Saiyam Pathak 演讲现场" src="https://project-hami.io/zh/assets/images/saiyam-pathak-56b0726f18661e34e76b5955ddcd15c6.jpg" width="1600" height="900" class="img_ev3q"></p>
<p>这场分享的要点在于：AI Factory 不只是把更多 GPU 集中起来，而是通过软件平台将硬件组织为可管理的计算环境，其中 Kubernetes 负责工作负载编排，GPU 共享让有限的加速计算资源服务更多团队。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="从云原生到-ai-原生">从云原生到 AI 原生<a href="https://project-hami.io/zh/blog/containerdays-singapore-2026-recap#%E4%BB%8E%E4%BA%91%E5%8E%9F%E7%94%9F%E5%88%B0-ai-%E5%8E%9F%E7%94%9F" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>Xu Chen（DaoCloud）讨论了东南亚数字化转型中从云原生应用现代化走向 AI 原生平台建设的路径，内容涵盖开放式 Cloud-Native AI 平台与开源生态的作用。HAMi 作为 Kubernetes 中异构计算设备共享与管理的项目被纳入这一技术版图，与其他云原生组件共同支撑 AI 工作负载。</p>
<p><img decoding="async" loading="lazy" alt="Xu Chen 演讲现场" src="https://project-hami.io/zh/assets/images/xu-chen-b6a6b97729fc63bd0f6759a12fb3a7c2.jpg" width="1600" height="1200" class="img_ev3q"></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="共同的主线">共同的主线<a href="https://project-hami.io/zh/blog/containerdays-singapore-2026-recap#%E5%85%B1%E5%90%8C%E7%9A%84%E4%B8%BB%E7%BA%BF" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>五场分享切入的角度不同，但反复回到同样的问题：资源如何分配、运行时如何执行限制、调度策略是否适合工作负载，以及平台能否同时观察资源的分配情况与实际使用情况。</p>
<p>GPU 共享不是所有工作负载的统一答案，切分粒度也并非越细越好。选择资源分配方式，需要结合任务特点、性能目标、隔离边界与平台整体利用率综合判断。</p>
<p>HAMi 社区将继续围绕 GPU 共享、异构算力管理、Kubernetes 调度与运行时隔离开展探索。欢迎访问<a class="" href="https://project-hami.io/zh/community">社区页面</a>或 <a href="https://github.com/Project-HAMi/HAMi" target="_blank" rel="noopener noreferrer" class="">HAMi 仓库</a>参与协作。</p>]]></content>
        <author>
            <name>HAMi 社区</name>
        </author>
        <category label="ContainerDays" term="ContainerDays"/>
        <category label="GPU" term="GPU"/>
        <category label="Kubernetes" term="Kubernetes"/>
        <category label="AI" term="AI"/>
        <category label="Singapore" term="Singapore"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[从 HAMi 到 HAMi-DRA：异构算力资源管理的演进]]></title>
        <id>https://project-hami.io/zh/blog/hami-dra-npu-sharing</id>
        <link href="https://project-hami.io/zh/blog/hami-dra-npu-sharing"/>
        <updated>2026-10-09T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[HAMi DRA 是什么、为什么需要它、如何设计：回顾 Device Plugin 时代 HAMi 的四大瓶颈（配额计量、抢占兼容、调度性能、死锁），解析 HAMi-DRA 的 Webhook 转换、DRA Driver 生态时间线与 HAMi-core 的保留角色，并给出选型建议与真实集群中的运行示例。]]></summary>
        <content type="html"><![CDATA[<p>2026 年 9 月 15 日，HAMi 社区发布了 <a href="https://github.com/Project-HAMi/HAMi-DRA" target="_blank" rel="noopener noreferrer" class="">HAMi DRA</a> 0.2.3。这个版本本身是一次常规迭代，但时间点值得注意：HAMi 2.9 版本直播宣布 HAMi-DRA 进入生产可用，NPU 的 DRA 支持计划随 2.10 发布。对关注异构算力调度的用户来说，HAMi DRA 已经从“实验性方向”变成了“可以认真评估的选项”。</p>
<p>围绕它的讨论也一直没有停。自从 Kubernetes 1.34 将 DRA（Dynamic Resource Allocation）推进至 GA，“DRA 会不会让 HAMi 过时”就成了社区频道里的高频问题；<a class="" href="https://project-hami.io/zh/blog/does-kubernetes-dra-replace-hami">《Kubernetes DRA 会取代 HAMi 吗？》</a>给出的回答是：DRA 吸收的是“请求与调度”的那一半，而“容器内强制执行”的那一半，DRA 从未设计去做，这正是 HAMi 保留的位置。</p>
<p>HAMi DRA 就是社区对这个争论的工程回应：一个迁移层，把存量 workload 里的 HAMi 风格资源请求自动转换成原生 ResourceClaim，调度与记账交还 kube-scheduler，运行时隔离继续由 HAMi-core 完成，业务侧一行代码不用改。当前 0.2.3 版本覆盖 NVIDIA GPU；在昇腾 DRA Driver 的配合下，这条链路已在真实的 310P3 集群上完成端到端验证，社区同步发布了配套的 <a class="" href="https://project-hami.io/zh/tutorials/labs/ascend-hami-dra">实验 21：用 HAMi DRA 共享昇腾 NPU</a>，提供命令与真实采集输出。本文系统介绍 HAMi DRA 的动机、设计、使用方式与当前边界。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="hami-解决了什么又卡在哪里">HAMi 解决了什么，又卡在哪里<a href="https://project-hami.io/zh/blog/hami-dra-npu-sharing#hami-%E8%A7%A3%E5%86%B3%E4%BA%86%E4%BB%80%E4%B9%88%E5%8F%88%E5%8D%A1%E5%9C%A8%E5%93%AA%E9%87%8C" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>HAMi 解决的问题很直接：Device Plugin 的 Extended Resource 只能表达整数张卡（<code>nvidia.com/gpu: 1</code>、<code>huawei.com/Ascend310P: 1</code>），而共享要的是“一块卡的一部分显存加一部分算力”：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># Device Plugin 只能这样申请：整卡，无法切分</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">limits</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">nvidia.com/gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 共享真正需要的是这样（HAMi 语法）：</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">limits</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">huawei.com/Ascend310P</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">huawei.com/Ascend310P-memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"8192"</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 8 GiB 显存</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">huawei.com/Ascend310P-core</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"50"</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 50% 算力</span><br></div></code></pre></div></div>
<p>为此 HAMi 在旧 API 之上搭起了完整体系：mutating webhook 改写 Pod、调度器扩展（extender）做共享调度、device plugin 上报与挂载、HAMi-core（<code>libvgpu</code> / <code>libvnpu</code>）在容器内强制执行配额。这套体系经受了生产检验：顺丰最高节省 57% GPU，SNOW 降本 55%，招商银行借拓扑感知调度实现硬件资源池 100% 利用。</p>
<p>但这些组件都是补在旧 API 上的，补得越多，结构性问题越明显。HAMi 现有方案最常见的四个痛点：</p>
<table><thead><tr><th style="text-align:left">痛点</th><th style="text-align:left">解释</th></tr></thead><tbody><tr><td style="text-align:left">配额管不住</td><td style="text-align:left">原生 ResourceQuota 能统计 <code>huawei.com/*</code> 这类 Extended Resource 的请求量，但只能按命名空间记总账，看不到 HAMi 在每块卡上做的内存加算力切分，想按团队限制真实用量，就得自己再造一套配额系统</td></tr><tr><td style="text-align:left">抢占用不上</td><td style="text-align:left">抢占（Preemption）等调度功能不认识树外调度器替它做的决定，高优先级任务来了，也没法让低优先级任务让位</td></tr><tr><td style="text-align:left">越忙越慢</td><td style="text-align:left">多个 Pod 同时调度到同一台节点时只能排队一个个来（节点锁），集群越忙，创建 Pod 越慢</td></tr><tr><td style="text-align:left">偶尔卡死</td><td style="text-align:left">一次失败的请求可能让整台节点被锁 5 分钟，期间这台节点上的新 Pod 全部干等</td></tr></tbody></table>
<p>这四点都不是代码写错了，而是路本身的问题：在“按节点整卡分配”的模型上模拟精细切分，这些代价躲不开。</p>
<p><img decoding="async" loading="lazy" alt="HAMi 在 Device Plugin 之上卡住的位置" src="https://project-hami.io/zh/assets/images/where-hami-stuck-zh-008a839caf160463407db3853147fcb4.png" width="2346" height="2168" class="img_ev3q"></p>
<p>图中左侧是 HAMi 为实现共享而自建的整条链路：从 Pod 到加速器之间的每一层都由 HAMi 自己实现。四个瓶颈的位置如图中虚线所示：第 1 个出在资源声明本身（Extended Resource 进原生配额的只有整卡计数，没有 HAMi 的精细切分），第 2、3、4 个都出在 Scheduler Extender 这一环。抢占受限是因为 HAMi 未实现 Extender API 中的可选抢占接口（preempt verb）、节点锁把并发退化为串行、失败请求可能锁住节点 5 分钟。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="dra-改变了什么">DRA 改变了什么<a href="https://project-hami.io/zh/blog/hami-dra-npu-sharing#dra-%E6%94%B9%E5%8F%98%E4%BA%86%E4%BB%80%E4%B9%88" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>Kubernetes 1.34 将 DRA（Dynamic Resource Allocation）核心 API 推进至 GA，集群设备资源管理从以节点为中心的分配模型，演进为以资源对象为核心的声明式模型。新 API 一共四个对象：</p>
<ul>
<li class=""><strong>ResourceSlice</strong>：设备清单。每块加速器是一个 device，携带属性（uuid、型号、PCIe 等）与容量（显存、算力）。</li>
<li class=""><strong>ResourceClaim</strong>：Pod 的设备需求声明，在 Pod 创建之前完成表达与约束。</li>
<li class=""><strong>DeviceClass</strong>：设备过滤与配置模板。</li>
<li class=""><strong>ResourceClaimTemplate</strong>：供 StatefulSet 等控制器批量生成 claim。</li>
</ul>
<p>对共享调度最关键的是 <strong>Consumable Capacity</strong>（KEP-5075）：设备以两个独立维度（显存、算力百分比）发布容量，<code>allowMultipleAllocations</code> 允许同一设备被多个 claim 消费，kube-scheduler 逐维度对账已消费容量，每次分配带独立 shareID。这意味着“多个 workload 合理消费同一加速器的部分容量”第一次成为调度器的原生能力，不再需要树外组件模拟。注意该 feature gate 在 1.34/1.35 需手动开启，1.36 起默认开启。</p>
<p>设备与分配在真实集群里的样子（节选）：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># ResourceSlice：每块 310P3 一个 device，发布两个维度的容量</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">devices</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> npu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">0</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token number" style="color:rgb(247, 140, 108)">0</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">attributes</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">productName</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain"> </span><span class="token key atrule">string</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> 310P3 </span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">uuid</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain"> </span><span class="token key atrule">string</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> 68496E64</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">20E05477</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">92C31323</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">6E78030A</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">BD003019 </span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">capacity</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">cores</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain"> </span><span class="token key atrule">value</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"100"</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain"> </span><span class="token key atrule">value</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> 21525Mi </span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">allowMultipleAllocations</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token boolean important" style="color:rgb(255, 88, 116)">true</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># ResourceClaim：申请其中一部分</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">capacity</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">requests</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">cores</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"50"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"8589934592"</span><br></div></code></pre></div></div>
<p><img decoding="async" loading="lazy" alt="DRA 对象模型与逐维度记账" src="https://project-hami.io/zh/assets/images/dra-model-zh-ab9ae62fd822084611f226eb714e0229.png" width="3716" height="2622" class="img_ev3q"></p>
<p>图中 Claim A 与 Claim B 各自申请同一块设备（npu-0-0）的一部分容量，kube-scheduler 对照 ResourceSlice 逐维度记账；容量记满后，Claim C 只能停在 pending，直到有 Pod 释放容量。</p>
<p>传统 Device Plugin 接口只能暴露简单容量信息，调度器“调度到节点后才发现资源不够”；DRA 让设备属性在调度阶段就可精确匹配。这里的迁移逻辑很清楚：瓶颈不在 HAMi 的调度算法，而在它脚下那层 API。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="hami-dra-的设计">HAMi DRA 的设计<a href="https://project-hami.io/zh/blog/hami-dra-npu-sharing#hami-dra-%E7%9A%84%E8%AE%BE%E8%AE%A1" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>HAMi DRA 没有推倒重来，而是只替换了上半部分：调度和记账改用 Kubernetes 原生实现，运行时隔离继续用 HAMi-core。</p>
<p><img decoding="async" loading="lazy" alt="HAMi DRA 请求路径：从 YAML 到加速器" src="https://project-hami.io/zh/assets/images/hami-dra-design-zh-1a3a2bf92a676bade93f60926a1d2fd2.png" width="3528" height="2888" class="img_ev3q"></p>
<p>整个过程从一份 YAML 开始。兼容模式的用户继续写 <code>huawei.com/Ascend310P</code> 这类资源名，HAMi DRA 的 Mutating Webhook 会在准入阶段把它转换成 ResourceClaim；原生模式的用户直接提交 ResourceClaim，跳过这一步。接下来 kube-scheduler 依据 ResourceSlice 完成调度和记账，把 Pod 绑定到节点。节点上的 kubelet 随即调用 DRA Driver 的 Prepare 接口，由它生成 CDI spec；containerd 按照 spec 创建容器，HAMi-core（<code>libvgpu</code> / <code>libvnpu</code>）也在这一步被注入容器、负责执行配额。workload 最终运行在 GPU、NPU、DCU 等加速器上，图中的编号 ① 到 ⑧ 对应这条链路的先后顺序。</p>
<p>三个组件，各管一段：<strong>HAMi-DRA Webhook</strong> 负责“怎么声明”，<strong>DRA driver</strong> 负责“怎么分配设备”，<strong>HAMi-core</strong> 负责“怎么共享设备”。</p>
<p><strong>Webhook</strong> 做两件事。一是多维度资源自动转换：把 HAMi 定义的多维度 ResourceName（如 <code>huawei.com/Ascend310P</code> 加 <code>-memory</code> 加 <code>-core</code>，NVIDIA 侧 <code>nvidia.com/gpu</code> 加 <code>gpumem</code> 加 <code>gpucores</code>）自动转换成 ResourceClaim 的 <code>count</code> 与 <code>capacity.requests</code>；HAMi 的选卡注解（<code>use-*-uuid</code> 等）转换成 CEL 选择器。二是 ResourceClaim 生命周期管理：claim 的创建与 Pod 绑定，Pod 删除时自动清理。注意 HAMi-DRA 自身不再包含调度器组件，原生 kube-scheduler、Volcano 等都能配合工作。</p>
<p><strong>DRA driver</strong> 是节点侧实现：发现设备并发布 ResourceSlice（属性加容量），处理 kubelet 的 <code>NodePrepareResources</code> 注入环境变量、创建共享目录，通过 <code>UnprepareResourceClaims</code> 在合适时机清理 HAMi-core 引入的临时文件，这也是设备生命周期管理比 Device Plugin 时代更完善的地方。设备注入走 CDI 标准接口落到 containerd。</p>
<p><strong>HAMi-core</strong> 原样保留：DRA 只解决“请求与调度”那一半，“容器内强制执行”那一半（显存配额、算力时间片）仍由 <code>libvgpu</code> / <code>libvnpu</code> 完成。这正是 <a class="" href="https://project-hami.io/zh/blog/does-kubernetes-dra-replace-hami">《Kubernetes DRA 会取代 HAMi 吗？》</a>的结论在架构上的落点。</p>
<p>调度性能的提升同样来自这次替换：借助 DRA 自身的调度系统，HAMi-DRA 避免了节点锁带来的性能衰退，同节点并发调度多个 Pod 时，原方案的 Pod 创建时间随并发近似线性劣化，社区内部测试中 HAMi-DRA 明显更低。这主要得益于 DRA 的资源预绑定机制：资源分配在调度阶段就已确定，减少了调度冲突与重试。整体可以概括为四组对照：NodeLock 到 No Locks，ResourceName 到 ResourceClaim，Device Plugin 到 DRA Driver，Annotation 到 Attributes。</p>
<p>还有一个容易被低估的变化是可观测性。传统模型里，资源信息来自 Node、使用情况来自 Pod，完整的资源视图需要聚合推断才能得到；DRA 模型里 ResourceSlice 描述设备清单、ResourceClaim 描述分配，资源视角本身就是一等公民。可观测性从“推断”变成了“直接建模”：运维可以直接从 ResourceClaim 看到每块加速器被谁占用、分配了多少显存、还剩多少余量，而不必从节点状态和 Pod 配置里反推。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="两种使用模式">两种使用模式<a href="https://project-hami.io/zh/blog/hami-dra-npu-sharing#%E4%B8%A4%E7%A7%8D%E4%BD%BF%E7%94%A8%E6%A8%A1%E5%BC%8F" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>DRA 的能力升级有一面经常被忽视的代价：写法复杂度，也就是常被提到的“UX 退化”。Device Plugin 的写法是一行：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">limits</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">nvidia.com/gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><br></div></code></pre></div></div>
<p>DRA 原生写法则是独立的 ResourceClaim 对象，<code>count</code>、<code>capacity.requests</code> 一层层嵌套，还要配上按设备属性过滤的 CEL 选择器：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">spec</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">devices</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">requests</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">exactly</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">allocationMode</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> ExactCount</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">count</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">capacity</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">            </span><span class="token key atrule">requests</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">              </span><span class="token key atrule">memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> 4194304k</span><br></div></code></pre></div></div>
<p>对已在使用 Device Plugin 的企业，迁移成本不只是改 YAML，而是整个团队要学一套新的资源声明范式。HAMi-DRA 对此的回答是两种提交方式，底层调度与切分逻辑一致：</p>
<ul>
<li class=""><strong>DRA 原生模式</strong>：手动创建 ResourceClaim 声明显存与算力，Pod 通过 <code>resourceClaims</code> 引用。适合新业务，能直接使用 CEL 选择器表达设备属性约束（例如拓扑感知、NUMA 亲和；相同机制已扩展到网卡等资源，配合 <code>firstAvailable</code> 还能表达多种候选分配方案）。</li>
<li class=""><strong>DevicePlugin 兼容模式</strong>：沿用传统 <code>nvidia.com/gpu</code> / <code>huawei.com/Ascend310P</code> 资源申请语法，Webhook 自动拦截并转换为 ResourceClaim。存量业务零改造迁移，这就是“无感迁移”的由来：这个设计把 DRA 从“专家接口”变成了“普通用户接口”。</li>
</ul>
<p>兼容模式的转换发生在准入阶段：</p>
<p><img decoding="async" loading="lazy" alt="兼容模式：Webhook 在准入阶段完成转换" src="https://project-hami.io/zh/assets/images/webhook-conversion-zh-1120f8fd6a1a3419c9ee3a74ab42448c.png" width="2810" height="870" class="img_ev3q"></p>
<p>Webhook 在准入阶段把三个 HAMi 资源条目换成 ResourceClaim 的 <code>count</code>、<code>capacity.requests</code>（8192 MiB 换算为字节数）与 CEL 选择器；claim 的生命周期与 Pod 绑定，Pod 删除时一并回收。</p>
<p>以一台昇腾 310P 真实集群里提交的 Pod 为例，<code>resources.limits</code> 里的三个 <code>huawei.com/*</code> 条目被替换为 <code>resources.claims</code> 与 <code>spec.resourceClaims</code>，生成的 ResourceClaim 中 HAMi 语义逐项进入 DRA 语义：</p>
<table><thead><tr><th style="text-align:left">ResourceClaim 字段</th><th style="text-align:left">来源</th><th style="text-align:left">转换规则</th></tr></thead><tbody><tr><td style="text-align:left"><code>count: 1</code></td><td style="text-align:left"><code>Ascend310P: 1</code></td><td style="text-align:left">整数直传</td></tr><tr><td style="text-align:left"><code>capacity.requests.memory: 8589934592</code></td><td style="text-align:left"><code>-memory: 8192</code>（MiB）</td><td style="text-align:left">MiB 转字节（8192 × 1024 × 1024）</td></tr><tr><td style="text-align:left"><code>capacity.requests.cores: "50"</code></td><td style="text-align:left"><code>-core: 50</code></td><td style="text-align:left">百分比直传</td></tr><tr><td style="text-align:left">CEL 选择器</td><td style="text-align:left"><code>use-Ascend310P-uuid</code> 注解</td><td style="text-align:left"><code>uuid in ["..."]</code>，值来自 ResourceSlice</td></tr></tbody></table>
<p>两个 Pod 各请求 8192 MiB 显存加 50 算力、指向同一块卡时，两个 claim 落到同一个 device 上，各自带独立 shareID，调度器在分配第二个之前已把第一个的消费记账。算力记满后（两个 claim 占满 100/100 cores，显存尚有余量），下一个相同请求被调度器直接拒绝（claim 停在 pending），Pod 删除后 claim 自动回收、记账恢复。容器内的强制执行依旧是 HAMi-core：<code>libvnpu.so</code> 拦截超限的显存申请并转为容器内 OOM，容器视角的设备被虚拟化为申请的 8 GiB 而非整卡 21 GiB。完整过程见<a class="" href="https://project-hami.io/zh/tutorials/labs/ascend-hami-dra">实验 21</a>。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="设备支持的时间线与为什么是现在">设备支持的时间线，与“为什么是现在”<a href="https://project-hami.io/zh/blog/hami-dra-npu-sharing#%E8%AE%BE%E5%A4%87%E6%94%AF%E6%8C%81%E7%9A%84%E6%97%B6%E9%97%B4%E7%BA%BF%E4%B8%8E%E4%B8%BA%E4%BB%80%E4%B9%88%E6%98%AF%E7%8E%B0%E5%9C%A8" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>HAMi DRA 的能力边界很大程度上取决于各厂商 DRA driver 的适配进度：</p>
<ul>
<li class="">2025.09：NVIDIA DRA Driver，Consumable Capacity 就绪</li>
<li class="">2026.03：海光 DCU DRA Driver 完成与 HAMi DRA 的集成</li>
<li class="">2026.04：Enflame（燧原）DRA Driver 集成推进中</li>
</ul>
<p>昇腾链路的打通，意味着 HAMi DRA 的能力边界扩展到了 NPU：<a class="" href="https://project-hami.io/zh/tutorials/labs/ascend-hami-dra">实验 21</a> 已在一台 310P3 服务器上，用官方 <a href="https://github.com/Project-HAMi/ascend-dra-driver" target="_blank" rel="noopener noreferrer" class="">Project-HAMi/ascend-dra-driver</a>（chart 0.1.1）把 HAMi 请求到 NPU 共享的完整链路跑通（实验所用镜像为修复 uuid 生成问题的开发构建，正式发布前行为可能变化）。HAMi 2.10.0 已于 2026 年 8 月发布，release notes 中已把昇腾 DRA 指向该 driver 仓库；driver 本身仍是预发布状态，现在仍是尝鲜与反馈的窗口。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="落地前的现实约束">落地前的现实约束<a href="https://project-hami.io/zh/blog/hami-dra-npu-sharing#%E8%90%BD%E5%9C%B0%E5%89%8D%E7%9A%84%E7%8E%B0%E5%AE%9E%E7%BA%A6%E6%9D%9F" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>落地阻力可以归纳为四条：</p>
<ul>
<li class=""><strong>高版本依赖</strong>：Consumable Capacity 依赖 Kubernetes 1.34+，其他 DRA 特性可能要求更高版本，生产环境推动升级不轻松。</li>
<li class=""><strong>有限的厂商支持</strong>：开箱即用的 DRA Driver 数量还少，支持的异构设备有限。</li>
<li class=""><strong>Scoring 缺失</strong>：设备调度层面 DRA 的打分能力尚缺，对特性化调度要求的支持有限。</li>
<li class=""><strong>学习成本</strong>：DRA 引入的新概念对集群部署与维护同学有门槛，HAMi 社区的对策正是用 Webhook 兼容模式把这份成本挡在用户视线之外。</li>
</ul>
<p>选型上，社区给出的建议：标准化集群、追求最优调度效果（如拓扑感知），推荐原生 HAMi；高度定制化（已有自己的调度器）或需要即插即用设备复用的集群，推荐 HAMi DRA。此外两条运维事实值得记住：DRA 模式与传统模式是同一请求的两条不同路径，一个请求不能同时走两条路；集群里仍装有 HAMi core 时，用实验 21 中的 webhook 豁免可以让两者共存于同一集群。使用 DRA 模式仍然会经过 HAMi-core。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="展望">展望<a href="https://project-hami.io/zh/blog/hami-dra-npu-sharing#%E5%B1%95%E6%9C%9B" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>HAMi-DRA 的 roadmap 上排着：更多异构设备（MetaX、天数智芯 Iluvatar CoreX）、更多 DRA 特性适配（List Attributes、Partitionable Devices）、兼容 Kubernetes 标准设备属性（<code>resource.kubernetes.io/pcieRoot</code>），以及与 Volcano、Kueue 的调度器集成。</p>
<p>再往大处看：Kubernetes 正在演进为 AI 基础设施的控制平面，而 HAMi 的定位是面向 Kubernetes 的加速器资源层，向下适配异构设备，向上支撑训练、推理与 Agent 等 workload，中间提供调度、虚拟化与资源抽象。HAMi-DRA 正是把这层资源能力与 Kubernetes 原生模型对齐的关键一步：HAMi 的价值收敛到运行时强制执行与异构生态适配，资源表达与调度则交给原生模型。如果你正在评估从 Device Plugin 模式迁移到 DRA，或在为异构集群寻找统一的资源管理层，欢迎试用 HAMi DRA 并向社区反馈。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="延伸阅读">延伸阅读<a href="https://project-hami.io/zh/blog/hami-dra-npu-sharing#%E5%BB%B6%E4%BC%B8%E9%98%85%E8%AF%BB" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class="">HAMi DRA 0.2.3 发布：<a href="https://github.com/Project-HAMi/HAMi-DRA/releases/tag/hami-dra-0.2.3" target="_blank" rel="noopener noreferrer" class="">Release hami-dra-0.2.3</a></li>
<li class="">杨守仁 AICon 深圳 2026 分享：<a href="https://aicon.infoq.cn/2026/shenzhen/presentation/7168" target="_blank" rel="noopener noreferrer" class="">从 HAMi 到 HAMi-DRA：异构环境的算力资源管理实践</a></li>
<li class="">王纪飞、James Deng KCD Beijing 2026 分享回顾：<a href="https://dynamia.ai/zh/blog/kcd-beijing-2026-dra-gpu-scheduling" target="_blank" rel="noopener noreferrer" class="">从 Device Plugin 到 DRA：GPU 调度范式升级与 HAMi-DRA 实践</a></li>
<li class="">HAMi 2.9 版本直播回顾（李孟轩）：<a href="https://dynamia.ai/zh/blog/hami-2.9-webinar-recap" target="_blank" rel="noopener noreferrer" class="">HAMi 2.9 昇腾软切分与 DRA 实战详解</a></li>
<li class="">社区上手指南：<a href="https://dynamia.ai/zh/blog/hami-dra-quickstart" target="_blank" rel="noopener noreferrer" class="">HAMi 正式接入 Kubernetes DRA：下一代 GPU 资源模型实践指南</a></li>
<li class="">Mesut Oezdil：<a class="" href="https://project-hami.io/zh/blog/does-kubernetes-dra-replace-hami">Kubernetes DRA 会取代 HAMi 吗？</a>（<a href="https://www.cncf.io/blog/2026/08/07/does-kubernetes-dra-replace-hami/" target="_blank" rel="noopener noreferrer" class="">英文原文</a>，CNCF 博客）</li>
<li class="">用户指南：<a class="" href="https://project-hami.io/zh/docs/installation/how-to-use-hami-dra">如何使用 HAMi DRA</a>；动手实验：<a class="" href="https://project-hami.io/zh/tutorials/labs/ascend-hami-dra">实验 21：用 HAMi DRA 共享昇腾 NPU</a>、<a class="" href="https://project-hami.io/zh/tutorials/labs/hami-dra">实验 4：用 DRA 切分 GPU</a></li>
<li class="">相关组件：<a href="https://github.com/Project-HAMi/HAMi-DRA" target="_blank" rel="noopener noreferrer" class="">Project-HAMi/HAMi-DRA</a> · <a href="https://github.com/Project-HAMi/ascend-dra-driver" target="_blank" rel="noopener noreferrer" class="">Project-HAMi/ascend-dra-driver</a> · <a href="https://github.com/Project-HAMi/hami-vnpu-core" target="_blank" rel="noopener noreferrer" class="">Project-HAMi/hami-vnpu-core</a></li>
</ul>]]></content>
        <author>
            <name>宋净超（Jimmy Song）</name>
            <uri>https://github.com/rootsongjc</uri>
        </author>
        <category label="HAMi" term="HAMi"/>
        <category label="DRA" term="DRA"/>
        <category label="Ascend" term="Ascend"/>
        <category label="NPU 共享" term="NPU 共享"/>
        <category label="Kubernetes" term="Kubernetes"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[KubeCon China 2026 回顾：两场 Keynote、一场闪电演讲，与一项案例大奖]]></title>
        <id>https://project-hami.io/zh/blog/kubecon-china-2026-recap</id>
        <link href="https://project-hami.io/zh/blog/kubecon-china-2026-recap"/>
        <updated>2026-09-10T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[HAMi 以 CNCF 孵化项目身份首次参加 KubeCon China：两场 Keynote、一场闪电演讲、一场万卡规模生产实践、全天值守的展台 T-1，招商银行采用 HAMi 的案例还斩获 Cloud Native China 2026 案例大奖。]]></summary>
        <content type="html"><![CDATA[<p>9 月 7-9 日，<a href="https://www.lfopensource.cn/kubecon-cloudnativecon-openinfra-summit-pytorch-conference-china/" target="_blank" rel="noopener noreferrer" class="">KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026</a>在上海国际会议中心落下帷幕。这是 HAMi<a class="" href="https://project-hami.io/zh/blog/hami-cncf-incubating">晋级 CNCF 孵化项目</a>后首次参加 KubeCon China：社区带来了两场 Keynote、一场闪电演讲、一场专场分享和一个全天值守的展台，外加 Keynote 舞台上揭晓的一项案例大奖。</p>
<p>四场分享的幻灯片均可在 <a class="" href="https://project-hami.io/zh/landing/kubecon-china">KubeCon China 2026 活动页</a>下载。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="keynote-1大规模运行前沿智能">Keynote 1：大规模运行前沿智能<a href="https://project-hami.io/zh/blog/kubecon-china-2026-recap#keynote-1%E5%A4%A7%E8%A7%84%E6%A8%A1%E8%BF%90%E8%A1%8C%E5%89%8D%E6%B2%BF%E6%99%BA%E8%83%BD" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="张潇（密瓜智能联合创始人兼 CEO）在开幕 Keynote 舞台上，与 Linux 基金会 CTO Chris Aniszczyk 同台分享" src="https://project-hami.io/zh/assets/images/xiao-zhang-keynote-13103810cc2057722e7926ceb1b20531.jpg" width="1920" height="1079" class="img_ev3q"></p>
<p>9 月 8 日上午的开幕 Keynote 由 Chris Aniszczyk（CTO, Cloud and Infrastructure, The Linux Foundation）与张潇（密瓜智能联合创始人兼 CEO）共同带来：当模型构建完成、AI 真正走向生产，挑战转向如何让每一块 GPU 发挥价值。云原生技术正在成为 AI 基础设施的操作系统层。</p>
<p>张潇的分享把 GPU 共享放到了故事中心：借助 HAMi，一张 GPU 可以切分给多个 LLM 工作负载共享，同一套调度平面还能延伸到异构加速器：</p>
<p><img decoding="async" loading="lazy" alt="开幕 Keynote 幻灯片“GPU sharing with HAMi”：单张 GPU 切分后同时服务多个 LLM 工作负载，四周是 HAMi 从细粒度 vGPU 到可观测性的能力矩阵" src="https://project-hami.io/zh/assets/images/keynote-gpu-sharing-532efea3e2dafb827327f7b372985d25.png" width="2000" height="1125" class="img_ev3q"></p>
<p>生产数据页则给出了采用者的实测结果：招商银行单卡推理吞吐提升 46.7%（ResNet 50·batch 32）；驾驶仿真场景 GPU 工时减少 30%；训推一体流水线所需 GPU 减半；页面上的 GPU 基础设施已有 90% 由 HAMi 管理。</p>
<p><img decoding="async" loading="lazy" alt="Keynote 幻灯片“HAMi in production”：单卡推理吞吐 +46.7%、驾驶仿真 GPU 工时 -30%、训推流水线 GPU -50%、90% 的 GPU 基础设施由 HAMi 管理" src="https://project-hami.io/zh/assets/images/keynote-hami-production-c8013ba8365930f097924a00a4b4deef.png" width="2000" height="1125" class="img_ev3q"></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="keynote-2基于-llm-d-的多元-ai-加速器-pd-分离-vllm-部署">Keynote 2：基于 llm-d 的多元 AI 加速器 PD 分离 vLLM 部署<a href="https://project-hami.io/zh/blog/kubecon-china-2026-recap#keynote-2%E5%9F%BA%E4%BA%8E-llm-d-%E7%9A%84%E5%A4%9A%E5%85%83-ai-%E5%8A%A0%E9%80%9F%E5%99%A8-pd-%E5%88%86%E7%A6%BB-vllm-%E9%83%A8%E7%BD%B2" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="王纪飞与李孟轩在 Grand Ballroom II + III 进行 llm-d Keynote" src="https://project-hami.io/zh/assets/images/llm-d-keynote-f8f92ba292602741f71eb363380cd5f6.jpg" width="1920" height="1440" class="img_ev3q"></p>
<p>同日上午，王纪飞（HAMi Approver，密瓜智能）与李孟轩（密瓜智能联合创始人兼 CTO）登台带来 5 分钟 Keynote，介绍专注于在 Kubernetes 上构建分布式 LLM 推理的 CNCF 项目 <a href="https://github.com/llm-d/llm-d" target="_blank" rel="noopener noreferrer" class="">llm-d</a>。当推理集群不再只有 NVIDIA 一种 GPU，PD 分离（Prefill/Decode Disaggregation）架构如何在多元加速器上高效部署 vLLM？他们的答案是：把 HAMi 的异构 GPU 共享与调度能力带入 llm-d 的推理拓扑，让切分、共享与调度跨硬件架构运转。</p>
<p><img decoding="async" loading="lazy" alt="Keynote 幻灯片“Optimize LLM-D inference”：请求经 llm-d router 进入 Prefill 与 Decode 实例，每个实例运行在 HAMi 通过 MIG/MPS 切分的 GPU 上，调度层可选用 HAMi、Volcano 或 KAI Scheduler" src="https://project-hami.io/zh/assets/images/llm-d-architecture-1e21cffc8ddd8de0f1882206048639d8.png" width="1500" height="844" class="img_ev3q"></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="闪电演讲从静态切片到弹性-gpu用-hami-实现动态-mig">闪电演讲：从静态切片到弹性 GPU，用 HAMi 实现动态 MIG<a href="https://project-hami.io/zh/blog/kubecon-china-2026-recap#%E9%97%AA%E7%94%B5%E6%BC%94%E8%AE%B2%E4%BB%8E%E9%9D%99%E6%80%81%E5%88%87%E7%89%87%E5%88%B0%E5%BC%B9%E6%80%A7-gpu%E7%94%A8-hami-%E5%AE%9E%E7%8E%B0%E5%8A%A8%E6%80%81-mig" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>午前的闪电演讲中，王纪飞再次聚焦 MIG 的老问题：静态预切分要求运维在工作负载到来之前猜好整卡分区布局。HAMi 给出的路线是调度驱动的动态 MIG：先由调度器完成 Pod 放置，再由 device plugin 在节点上重配 MIG 实例，负载在运行时拿到自己的 MIG UUID。</p>
<blockquote>
<p>GPU partitioning should follow scheduling, not precede it.（GPU 分区应该跟随调度，而不是先于调度。）</p>
</blockquote>
<p><img decoding="async" loading="lazy" alt="闪电演讲幻灯片“Dynamic MIG — Publish capacity”：HAMi 调度器放置 Pod，device plugin 经 NVML 将 GPU 重切为 MIG 实例，负载在运行时获得 MIG UUID" src="https://project-hami.io/zh/assets/images/dynamic-mig-publish-capacity-b478b624f7c675b3708d290f0bb51832.png" width="1500" height="844" class="img_ev3q"></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="专场分享合合信息如何支撑数十亿次文档扫描">专场分享：合合信息如何支撑数十亿次文档扫描<a href="https://project-hami.io/zh/blog/kubecon-china-2026-recap#%E4%B8%93%E5%9C%BA%E5%88%86%E4%BA%AB%E5%90%88%E5%90%88%E4%BF%A1%E6%81%AF%E5%A6%82%E4%BD%95%E6%94%AF%E6%92%91%E6%95%B0%E5%8D%81%E4%BA%BF%E6%AC%A1%E6%96%87%E6%A1%A3%E6%89%AB%E6%8F%8F" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="李孟轩现场分享合合信息生产实践" src="https://project-hami.io/zh/assets/images/intsig-session-adb9387565b654905ff19a03d53a0215.jpg" width="1920" height="1440" class="img_ev3q"></p>
<p><img decoding="async" loading="lazy" alt="Walter Duan（合合信息）在台上分享 IntSig 训推总览" src="https://project-hami.io/zh/assets/images/walter-duan-3d11cb56019dfe98250fb3e3341d7b3d.jpg" width="1920" height="1440" class="img_ev3q"></p>
<p>下午的专场分享由李孟轩（密瓜智能联合创始人兼 CTO）与 Walter Duan（合合信息）带来，深入剖析了一个多数团队梦寐以求的 GPU 集群：万卡规模的训推一体平台，自建数据中心与多云混合部署，运行 1000+ 在线推理服务与 1000+ 离线训练任务，训练侧 24 小时利用率保持在 90% 以上，训练算力覆盖 Hopper、Ampere、Blackwell 与昇腾。</p>
<p><img decoding="async" loading="lazy" alt="专场幻灯片“CaseStudy: IntSig 训推总览”：万卡集群、1000+ 在线推理服务、1000+ 离线训练任务、90%+ 24 小时利用率" src="https://project-hami.io/zh/assets/images/intsig-cluster-overview-bd92f59ba32dc59de2c7c10b1711496c.png" width="1500" height="844" class="img_ev3q"></p>
<p>合合信息从腾讯 QGPU 迁移到 HAMi 后，一站式获得虚拟化、调度与监控能力。部署打法：小模型推理用切片、高负载与大模型保留整卡；Binpack 优先装箱减少碎片；高负载与低负载混布；分配指标接入监控、从分配到使用形成闭环；配合 Karpenter 实现弹性扩容。实测收益：GPU 利用率提升 50%、综合成本降低 30%、推理性能下降控制在 10% 以内。</p>
<p><img decoding="async" loading="lazy" alt="专场幻灯片“IntSig HAMi 部署方式与收益”：五种部署模式与实测收益：GPU 利用率 +50%、成本 -30%、推理性能下降控制在 10% 以内" src="https://project-hami.io/zh/assets/images/intsig-hami-results-4ba2edc75f725aa942600cf99bf0c653.png" width="1500" height="844" class="img_ev3q"></p>
<p>演讲还集中披露了 HAMi 的生产案例数据：顺丰 GPU 从 1400 张精简到 1000 张、业务不受影响；招商银行 10000+ GPU、利用率 20% → 80%；蔚来 CI 效率提升 10 倍；工商银行 GPU 利用率 20% → 70%。同时也有“烧钱的反模式”警示：GPU 切分低于 1/6 时反而适得其反。结尾现场演示了合合信息开源的 AI 终端 Chaterm，用自然语言直接操作 GPU 集群。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="招商银行斩获案例大奖">招商银行斩获案例大奖<a href="https://project-hami.io/zh/blog/kubecon-china-2026-recap#%E6%8B%9B%E5%95%86%E9%93%B6%E8%A1%8C%E6%96%A9%E8%8E%B7%E6%A1%88%E4%BE%8B%E5%A4%A7%E5%A5%96" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="Keynote 大屏揭晓招商银行为 Cloud Native China 2026 案例大奖得主，其参考架构基于 Kubernetes、Kueue、KEDA、Fluid、Prometheus 与 HAMi 构建" src="https://project-hami.io/zh/assets/images/case-study-award-20f92585a13631aee6bdbef7eff5ddf8.jpg" width="1920" height="1439" class="img_ev3q"></p>
<p>还有一个值得记住的 Keynote 瞬间：招商银行被评为 Cloud Native China 2026 案例大奖得主，其参考架构组合了 Kubernetes、Kueue、KEDA、Fluid 与 Prometheus，GPU 共享层正是 HAMi，也就是那家把 10000+ GPU 利用率从 20% 提升到 80% 的银行。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="展台-t-1">展台 T-1<a href="https://project-hami.io/zh/blog/kubecon-china-2026-recap#%E5%B1%95%E5%8F%B0-t-1" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="HAMi 社区伙伴在 Grand Ballroom I 的 T-1 展台合影" src="https://project-hami.io/zh/assets/images/hami-booth-81715f9c70652d7a9ae0de7af45a0b47.jpg" width="1920" height="1440" class="img_ev3q"></p>
<p>维护者从上午一直值守到闭馆。聊得最多的话题：如何在不改动应用的前提下把 HAMi 引入现有集群；与 Volcano、Kueue、KAI Scheduler 的协同；MIG、vGPU 等切分方案的选型；以及 fleet 规模的多租户 GPU 管理。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="观看与下载">观看与下载<a href="https://project-hami.io/zh/blog/kubecon-china-2026-recap#%E8%A7%82%E7%9C%8B%E4%B8%8E%E4%B8%8B%E8%BD%BD" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>所有 session 均有录制，会后将在 CNCF 官方频道上线。四场分享（两场 Keynote、一场闪电演讲与合合信息专场）的幻灯片均可从 <a class="" href="https://project-hami.io/zh/landing/kubecon-china">KubeCon China 2026 活动页</a>下载。</p>
<p>欢迎加入 <a class="" href="https://project-hami.io/zh/community">HAMi 社区</a>参与讨论。</p>]]></content>
        <author>
            <name>HAMi 社区</name>
        </author>
        <category label="KubeCon" term="KubeCon"/>
        <category label="GPU" term="GPU"/>
        <category label="Kubernetes" term="Kubernetes"/>
        <category label="AI" term="AI"/>
        <category label="China" term="China"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[HAMi 可组合调度策略：mutex、binpack、spread、numa 如何协同工作]]></title>
        <id>https://project-hami.io/zh/blog/composable-scheduler-policies</id>
        <link href="https://project-hami.io/zh/blog/composable-scheduler-policies"/>
        <updated>2026-08-26T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[HAMi v2.10.0 让 hami.io/gpu-scheduler-policy 接受有序的、逗号分隔的策略列表。本文解释该特性背后的“先过滤、后排序”求值模型、各策略的语义、常用组合配方，以及上手与验证步骤。]]></summary>
        <content type="html"><![CDATA[<p>HAMi 一直通过 <code>hami.io/gpu-scheduler-policy</code> 注解提供按 Pod 生效的 GPU 调度策略：<code>binpack</code> 尽量把负载堆叠到少数卡上，<code>spread</code> 尽量打散，v2.10.0 新增的 <code>mutex</code> 则要求独占一张卡。但在 v2.10.0 之前，这个注解只能填<strong>一个</strong>值。</p>
<p>真实集群很少只需要一种行为。一份典型的生产诉求是这样的：推理副本要紧凑装箱，留出整卡空闲；同时每个 Pod 的多卡要落在同一 NUMA 节点上以保证带宽；而时延敏感的那批负载，干脆要独占几张卡。一句话里出现了<strong>三种</strong>策略。在 v2.10.0 之前，你只能选一个，放弃其余。</p>
<p>v2.10.0 补上了这块拼图：<code>hami.io/gpu-scheduler-policy</code> 现在接受<strong>有序的、逗号分隔的列表</strong>，过滤型与排序型策略可以组合（<a href="https://github.com/Project-HAMi/HAMi/pull/2621" target="_blank" rel="noopener noreferrer" class="">#2621</a>，<a href="https://github.com/mesutoezdil" target="_blank" rel="noopener noreferrer" class="">@mesutoezdil</a>，关闭 <a href="https://github.com/Project-HAMi/HAMi/issues/2010" target="_blank" rel="noopener noreferrer" class="">#2010</a>）。本文解释组合是如何求值的，以及如何上手与验证。如果你喜欢动手学习，配套的<a class="" href="https://project-hami.io/zh/tutorials/labs/composable-scheduler-policies-gke">实验 14：在 GKE 上验证可组合调度策略</a>会在真实集群上逐一复现下文的每个场景。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="两类策略过滤器与排序键">两类策略：过滤器与排序键<a href="https://project-hami.io/zh/blog/composable-scheduler-policies#%E4%B8%A4%E7%B1%BB%E7%AD%96%E7%95%A5%E8%BF%87%E6%BB%A4%E5%99%A8%E4%B8%8E%E6%8E%92%E5%BA%8F%E9%94%AE" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>理解这个特性的关键是认识到：HAMi 的 GPU 策略从来就不是同一种东西，它们分属两种角色：</p>
<table><thead><tr><th style="text-align:left">策略</th><th style="text-align:left">角色</th><th style="text-align:left">作用</th></tr></thead><tbody><tr><td style="text-align:left"><code>mutex</code></td><td style="text-align:left"><strong>过滤器</strong></td><td style="text-align:left">只有<strong>当前没有任何负载</strong>的卡才是候选，所有已被占用的卡都被筛掉。</td></tr><tr><td style="text-align:left"><code>topology-aware</code></td><td style="text-align:left"><strong>过滤器</strong>（仅 NVIDIA）</td><td style="text-align:left">只保留 NVLink/NVSwitch 拓扑满足多卡请求的卡，需要先启用拓扑感知调度。</td></tr><tr><td style="text-align:left"><code>binpack</code></td><td style="text-align:left"><strong>排序键</strong></td><td style="text-align:left">偏好<strong>使用率最高</strong>的合格卡，让负载集中。</td></tr><tr><td style="text-align:left"><code>spread</code></td><td style="text-align:left"><strong>排序键</strong></td><td style="text-align:left">偏好<strong>使用率最低</strong>的合格卡，让负载分散。</td></tr><tr><td style="text-align:left"><code>numa</code></td><td style="text-align:left"><strong>排序键</strong></td><td style="text-align:left">偏好 NUMA 节点编号更低的卡，提升 locality。</td></tr></tbody></table>
<p>过滤器回答的是每张卡一个是非题：这张卡到底能不能参与；排序键回答的是排序题：在合格卡中谁排第一。两种角色天然不冲突，因此可以组合：<strong>过滤器先运行、缩小候选集，排序键再对幸存者排序</strong>。</p>
<p>策略链表达的正是这件事：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 仅独占 GPU，再紧凑装箱，NUMA 作为 tiebreaker</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">metadata</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">annotations</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">hami.io/gpu-scheduler-policy</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"mutex,binpack,numa"</span><br></div></code></pre></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="调度器如何求值一条策略链">调度器如何求值一条策略链<a href="https://project-hami.io/zh/blog/composable-scheduler-policies#%E8%B0%83%E5%BA%A6%E5%99%A8%E5%A6%82%E4%BD%95%E6%B1%82%E5%80%BC%E4%B8%80%E6%9D%A1%E7%AD%96%E7%95%A5%E9%93%BE" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>HAMi 调度器在过滤和打分一个节点的 GPU 时，策略链按固定流水线求值：</p>
<figure class="mermaid-figure"><figcaption>HAMi 如何求值一条策略链</figcaption></figure>
<p>规则依次是：</p>
<ol>
<li class=""><strong>解析。</strong> 注解按逗号切分，忽略每一项两侧的空白，重复项被丢弃，只有已知的策略名会生效。</li>
<li class=""><strong>先过滤。</strong> <code>mutex</code> 和 <code>topology-aware</code> 在任何排序之前的过滤阶段生效。对带 <code>mutex</code> 的 Pod 来说，一张卡是否合格只看一个问题：这张卡当前是否一个租户都没有？哪怕卡上只跑着一个很小的负载、剩余显存再多，这张卡也会被整体排除，因此 <code>mutex</code> Pod 只能落到完全空闲的卡上。当所有卡都有租户时，Pod 会保持 Pending，原因为 <code>ExclusiveDeviceAllocateConflict</code>。</li>
<li class=""><strong>按书写顺序排序。</strong> <code>binpack</code>、<code>spread</code>、<code>numa</code> 构成一个有序的排序键列表。第一个键起决定作用：<code>binpack,numa</code> 先按 binpack 排序，只有当 binpack 给两张卡打出相同分数时，才用 NUMA 节点编号决胜负。写成 <code>numa,binpack</code> 则优先级相反，列表的顺序就是键的顺序。</li>
<li class=""><strong>确定性决胜。</strong> 如果两张卡在链上的所有键下都比较结果相同，设备索引更小者胜出。因此在空闲集群上，相同的请求会产生相同、可复现的放置结果。</li>
<li class=""><strong>仅含过滤器时回退。</strong> 一条完全不含排序键的链（例如 <code>"mutex,topology-aware"</code>）在过滤之后回退为 <code>spread</code> 排序。</li>
<li class=""><strong>单值行为不变。</strong> 单独的 <code>"binpack"</code>、<code>"spread"</code> 或 <code>"mutex"</code> 与 v2.10.0 之前的行为完全一致，升级时存量负载无需任何改动。</li>
</ol>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="排序键到底在比什么">排序键到底在比什么<a href="https://project-hami.io/zh/blog/composable-scheduler-policies#%E6%8E%92%E5%BA%8F%E9%94%AE%E5%88%B0%E5%BA%95%E5%9C%A8%E6%AF%94%E4%BB%80%E4%B9%88" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>对 <code>binpack</code> 和 <code>spread</code> 来说，“最满/最空”不是拍脑袋：调度器根据每张卡已分配的算力和显存计算利用率得分，权重由 slot、core、memory 三项组成（可以用 <code>hami.io/device-scoring-weights</code> 注解按 Pod 调整权重）。<code>binpack</code> 选<strong>分数最高</strong>的卡，<code>spread</code> 选<strong>分数最低</strong>的卡。<code>numa</code> 键比较的是 NVML 上报的每张卡所在 NUMA 节点编号。</p>
<p>值得一提的是 v2.10.0 同时修复了一个相关老问题（<a href="https://github.com/Project-HAMi/HAMi/pull/2012" target="_blank" rel="noopener noreferrer" class="">#2012</a>）：此前即使指定纯 <code>binpack</code>/<code>spread</code>，NUMA 也是<em>首要</em>排序键，导致负载无视实际负载地被钉死在一个 NUMA 节点上。修复后，利用率得分成为主键，NUMA 只做决胜，这正是 <code>numa</code> 在策略链中扮演的 tiebreaker 角色。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="实例一个-pod-在四张卡上的选择路径">实例：一个 Pod 在四张卡上的选择路径<a href="https://project-hami.io/zh/blog/composable-scheduler-policies#%E5%AE%9E%E4%BE%8B%E4%B8%80%E4%B8%AA-pod-%E5%9C%A8%E5%9B%9B%E5%BC%A0%E5%8D%A1%E4%B8%8A%E7%9A%84%E9%80%89%E6%8B%A9%E8%B7%AF%E5%BE%84" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>把上面的流程放进实验 14 的真实场景：一个节点、四块 Tesla T4，一个带 <code>mutex,binpack</code> 注解、申请 1 个 vGPU（1000 MiB 切片）的 Pod 待调度。下图每张卡标出了租户数与利用率得分，箭头就是选择路径：</p>
<figure class="mermaid-figure"><figcaption>一个 mutex,binpack Pod 在四张候选卡上的选择路径</figcaption></figure>
<p>从图中可以读出两件事：</p>
<ul>
<li class="">最忙的 GPU 0 根本没有进入排序步骤：<code>mutex</code> 过滤器先把它剔除了，尽管纯 <code>binpack</code> 会以 3.30 的最高分优先选它。过滤先于排序，所以组合链压根不会考虑纯 <code>binpack</code> 会选的那张卡。</li>
<li class="">两张幸存卡的得分相同（都是 0.00），<code>binpack</code> 无法区分，链继续落到设备索引决胜，确定性地选择 GPU 2。</li>
</ul>
<p>这些数字来自实测：实验中带一个 1000 MiB 租户的卡得分 <code>1.651042</code>，空闲卡得分 <code>0.000000</code>，均来自调度器的打分日志。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="常用配方">常用配方<a href="https://project-hami.io/zh/blog/composable-scheduler-policies#%E5%B8%B8%E7%94%A8%E9%85%8D%E6%96%B9" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<table><thead><tr><th style="text-align:left">注解</th><th style="text-align:left">含义</th></tr></thead><tbody><tr><td style="text-align:left"><code>"binpack"</code></td><td style="text-align:left">把该 Pod 堆到能放下的最忙的卡上，最大化整卡空闲。</td></tr><tr><td style="text-align:left"><code>"spread"</code>（默认）</td><td style="text-align:left">放到最空的卡上，最小化邻居间的争抢。</td></tr><tr><td style="text-align:left"><code>"mutex"</code></td><td style="text-align:left">要求一张当前零用户的卡（注意：之后调度的普通 Pod 仍可能加入这张卡）。</td></tr><tr><td style="text-align:left"><code>"binpack,numa"</code></td><td style="text-align:left">紧凑装箱；装箱打平时偏好 NUMA 编号更低的卡。</td></tr><tr><td style="text-align:left"><code>"mutex,binpack"</code></td><td style="text-align:left">独占且紧凑：在空闲卡中选最忙的一张，降低碎片化。</td></tr><tr><td style="text-align:left"><code>"mutex,binpack,numa"</code></td><td style="text-align:left">“时延层”配方：独占卡、空闲卡中紧凑装箱、NUMA 作为决胜。</td></tr></tbody></table>
<p>关于 <code>mutex</code> 语义的说明：它保证的是<strong>放置时刻</strong>这张卡没有用户。之后调度的普通 Pod 仍可能加入这张卡。若要让卡在负载整个生命周期内保持独占，请申请它的全部资源（显存和算力），或用 <code>nvidia.com/use-gpuuuid</code> 显式固定。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="三步上手">三步上手<a href="https://project-hami.io/zh/blog/composable-scheduler-policies#%E4%B8%89%E6%AD%A5%E4%B8%8A%E6%89%8B" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><strong>1. 升级到 HAMi v2.10.0 或更高版本。</strong></p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm repo </span><span class="token function" style="color:rgb(130, 170, 255)">add</span><span class="token plain"> hami-charts https://project-hami.github.io/HAMi/</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm repo update</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm upgrade hami hami-charts/hami </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> kube-system </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--version</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">2.10</span><span class="token plain">.0</span><br></div></code></pre></div></div>
<p>集群级默认值仍是节点选择 <code>binpack</code>、卡选择 <code>spread</code>（chart values 中的 <code>scheduler.defaultSchedulerPolicy.nodeSchedulerPolicy</code> / <code>gpuSchedulerPolicy</code>），升级不会改变任何默认值。</p>
<p><strong>2. 给需要组合行为的 Pod 打注解。</strong></p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">apiVersion</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> v1</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">kind</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Pod</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">metadata</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> inference</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">latency</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">tier</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">annotations</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">hami.io/gpu-scheduler-policy</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"mutex,binpack,numa"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">spec</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">containers</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> app</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">image</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> ubuntu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token number" style="color:rgb(247, 140, 108)">22.04</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">command</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">[</span><span class="token string" style="color:rgb(195, 232, 141)">"bash"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"-c"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"sleep 86400"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">]</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token key atrule">limits</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">nvidia.com/gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 1 个 vGPU</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">nvidia.com/gpumem</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">8000</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 显存配额 8000 MiB</span><br></div></code></pre></div></div>
<p><strong>3. 验证决策。</strong> 调度器会把选中的卡写到 Pod 上，因此无需进入容器即可验证放置结果：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 选中卡的设备 UUID、厂商、显存切片与算力百分比</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl get pod inference-latency-tier </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-o</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">jsonpath</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token string" style="color:rgb(195, 232, 141)">'{.metadata.annotations.hami\.io/vgpu-devices-allocated}'</span><span class="token punctuation" style="color:rgb(199, 146, 234)">;</span><span class="token plain"> </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">echo</span><br></div></code></pre></div></div>
<p>对于保持 Pending 的 <code>mutex</code> Pod，<code>kubectl describe pod</code> 会显示 <code>ExclusiveDeviceAllocateConflict</code> 原因；调度器在过滤时也会为每张候选卡输出一行利用率打分日志（<code>computer score is</code>），可以直接观察 <code>binpack</code> 与 <code>spread</code> 的排序依据：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> kube-system logs deploy/hami-scheduler </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-c</span><span class="token plain"> vgpu-scheduler-extender </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--tail</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">-1 </span><span class="token operator" style="color:rgb(137, 221, 255)">|</span><span class="token plain"> </span><span class="token function" style="color:rgb(130, 170, 255)">grep</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">'computer score'</span><span class="token plain"> </span><span class="token operator" style="color:rgb(137, 221, 255)">|</span><span class="token plain"> </span><span class="token function" style="color:rgb(130, 170, 255)">tail</span><br></div></code></pre></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="在真实集群上试试">在真实集群上试试<a href="https://project-hami.io/zh/blog/composable-scheduler-policies#%E5%9C%A8%E7%9C%9F%E5%AE%9E%E9%9B%86%E7%BE%A4%E4%B8%8A%E8%AF%95%E8%AF%95" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>读过滤器与排序键是一回事，亲眼看一个 <code>mutex</code> Pod 在你释放一张卡之前一直 Pending，再看 <code>mutex,binpack</code> 拒绝纯 <code>binpack</code> 本会选中的卡，是另一回事。<a class="" href="https://project-hami.io/zh/tutorials/labs/composable-scheduler-policies-gke">实验 14：在 GKE 上验证可组合调度策略</a>在一台挂四块 Tesla T4 的 GKE 节点上跑完整的特性矩阵：默认 <code>spread</code>、<code>binpack</code> 堆叠、<code>mutex</code> 阻塞与释放、组合的 <code>mutex,binpack</code> 行为，全部通过分配注解、事件和调度器日志验证。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="参考资料">参考资料<a href="https://project-hami.io/zh/blog/composable-scheduler-policies#%E5%8F%82%E8%80%83%E8%B5%84%E6%96%99" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class="">Pull request：<a href="https://github.com/Project-HAMi/HAMi/pull/2621" target="_blank" rel="noopener noreferrer" class="">feat: support comma-separated gpu-scheduler-policy combinations (#2621)</a></li>
<li class="">Issue：<a href="https://github.com/Project-HAMi/HAMi/issues/2010" target="_blank" rel="noopener noreferrer" class="">Composable scheduling policies (#2010)</a></li>
<li class=""><a class="" href="https://project-hami.io/zh/docs/developers/scheduling">调度策略设计文档</a></li>
<li class=""><a class="" href="https://project-hami.io/zh/docs/userguide/configure">配置参考</a></li>
<li class="">动手实验：<a class="" href="https://project-hami.io/zh/tutorials/labs/composable-scheduler-policies-gke">实验 14：在 GKE 上验证可组合调度策略</a></li>
</ul>]]></content>
        <author>
            <name>宋净超（Jimmy Song）</name>
            <uri>https://github.com/rootsongjc</uri>
        </author>
        <category label="HAMi" term="HAMi"/>
        <category label="调度" term="调度"/>
        <category label="GPU" term="GPU"/>
        <category label="Kubernetes" term="Kubernetes"/>
        <category label="云原生" term="云原生"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[HAMi 即将亮相 KubeCon China 2026：两场 Keynote、两场技术分享，展台 T-1 等你]]></title>
        <id>https://project-hami.io/zh/blog/hami-at-kubecon-china-2026</id>
        <link href="https://project-hami.io/zh/blog/hami-at-kubecon-china-2026"/>
        <updated>2026-08-25T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[HAMi 将以 CNCF 孵化项目的身份首次亮相 KubeCon China 2026（9 月 7-9 日·上海国际会议中心），带来两场 Keynote、一场闪电演讲、一场专场技术分享，并设置项目展台 T-1。从 PD 分离的异构推理部署，到 5 分钟讲透 MIG 动态切分，再到千卡规模的 GPU 虚拟化生产实践。]]></summary>
        <content type="html"><![CDATA[<p>9 月 7-9 日，<a href="https://www.lfopensource.cn/kubecon-cloudnativecon-openinfra-summit-pytorch-conference-china/" target="_blank" rel="noopener noreferrer" class="">KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026</a>将在上海国际会议中心举行。这也是 HAMi 在今年 7 月<a class="" href="https://project-hami.io/zh/blog/hami-cncf-incubating">晋级 CNCF 孵化（Incubating）项目</a>后，首次以孵化项目的身份亮相 KubeCon China。</p>
<p>本届大会上，HAMi 社区将带来<strong>两场 Keynote、一场闪电演讲、一场专场技术分享</strong>，并在现场设置<strong>项目展台（T-1）</strong>：从 PD 分离的异构推理部署，到 5 分钟讲透 MIG 动态切分，再到千卡规模的 GPU 虚拟化生产实践。欢迎社区的小伙伴到现场交流。</p>
<p><img decoding="async" loading="lazy" alt="KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026" src="https://project-hami.io/zh/assets/images/blog-hami-at-kubecon-china-2026-banner-0d77c04141cacaa84135989a858365a3.png" width="2410" height="846" class="img_ev3q"></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="大会信息">大会信息<a href="https://project-hami.io/zh/blog/hami-at-kubecon-china-2026#%E5%A4%A7%E4%BC%9A%E4%BF%A1%E6%81%AF" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class=""><strong>时间</strong>：2026 年 9 月 7-9 日（9 月 8-9 日为主会日）</li>
<li class=""><strong>地点</strong>：中国·上海国际会议中心</li>
<li class=""><strong>官网及报名</strong>：<a href="https://www.lfopensource.cn/kubecon-cloudnativecon-openinfra-summit-pytorch-conference-china/" target="_blank" rel="noopener noreferrer" class="">大会官网</a></li>
<li class=""><strong>完整议程</strong>：<a href="https://www.lfopensource.cn/kubecon-cloudnativecon-openinfra-summit-pytorch-conference-china/program/schedule/" target="_blank" rel="noopener noreferrer" class="">大会议程</a></li>
</ul>
<p>HAMi 的全部活动集中在 9 月 8 日：上午两场 Keynote，午间一场 Lightning Talk，下午一场生产实践 Session，展台则全天开放。Keynote 环节的完整阵容以大会官网最终公布的议程为准；所有 session 均会录制，会后将在 CNCF 官方频道上线。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="keynote-1大规模运行前沿智能">Keynote 1：大规模运行前沿智能<a href="https://project-hami.io/zh/blog/hami-at-kubecon-china-2026#keynote-1%E5%A4%A7%E8%A7%84%E6%A8%A1%E8%BF%90%E8%A1%8C%E5%89%8D%E6%B2%BF%E6%99%BA%E8%83%BD" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class=""><strong>原题</strong>：Operating Frontier Intelligence at Scale</li>
<li class=""><strong>时间</strong>：9 月 8 日 09:12-09:22</li>
<li class=""><strong>地点</strong>：Grand Ballroom II + III</li>
<li class=""><strong>讲者</strong>：Chris Aniszczyk（CTO, Cloud and Infrastructure, The Linux Foundation）、张潇（密瓜智能联合创始人兼 CEO）</li>
</ul>
<p>大会开幕首场 Keynote。Linux 基金会 CTO Chris Aniszczyk 将与张潇同台：当模型构建完成、AI 真正走向生产，挑战转向如何让每一块 GPU 发挥价值、随需求动态扩展、保持可靠性，并理解日益复杂的系统。云原生技术正在成为支撑这一切的操作系统层。演讲还将探讨可观测性在 AI 系统复杂化过程中与日俱增的作用，以及为什么下一代 AI 基础设施将依赖跨技术栈协同工作的开放技术。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="keynote-2基于-llm-d-的多元-ai-加速器-pd-分离-vllm-部署">Keynote 2：基于 llm-d 的多元 AI 加速器 PD 分离 vLLM 部署<a href="https://project-hami.io/zh/blog/hami-at-kubecon-china-2026#keynote-2%E5%9F%BA%E4%BA%8E-llm-d-%E7%9A%84%E5%A4%9A%E5%85%83-ai-%E5%8A%A0%E9%80%9F%E5%99%A8-pd-%E5%88%86%E7%A6%BB-vllm-%E9%83%A8%E7%BD%B2" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class=""><strong>原题</strong>：PD Disaggregation vLLM Deployment on Alternative AI Accelerators Using llm-d</li>
<li class=""><strong>时间</strong>：9 月 8 日 09:59-10:04</li>
<li class=""><strong>地点</strong>：Grand Ballroom II + III</li>
<li class=""><strong>讲者</strong>：王纪飞（HAMi Approver，密瓜智能）、李孟轩（密瓜智能联合创始人兼 CTO）</li>
</ul>
<p><a href="https://github.com/llm-d" target="_blank" rel="noopener noreferrer" class="">llm-d</a> 是专注于在 Kubernetes 上构建分布式 LLM 推理的 CNCF 项目。当推理集群里不再只有一种 GPU，PD 分离（Prefill/Decode Disaggregation）架构如何在非 NVIDIA 的多元加速器上高效部署 vLLM？这场 Keynote 将介绍 llm-d 对异构算力环境的支持：HAMi 将异构 GPU 共享与调度能力带入 llm-d 的推理拓扑，让切分、共享与调度跨硬件架构运转。5 分钟，干货密度拉满。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="闪电演讲从静态切片到弹性-gpu用-hami-实现动态-mig">闪电演讲：从静态切片到弹性 GPU，用 HAMi 实现动态 MIG<a href="https://project-hami.io/zh/blog/hami-at-kubecon-china-2026#%E9%97%AA%E7%94%B5%E6%BC%94%E8%AE%B2%E4%BB%8E%E9%9D%99%E6%80%81%E5%88%87%E7%89%87%E5%88%B0%E5%BC%B9%E6%80%A7-gpu%E7%94%A8-hami-%E5%AE%9E%E7%8E%B0%E5%8A%A8%E6%80%81-mig" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class=""><strong>原题</strong>：From Static Slices to Elastic GPUs: Dynamic MIG with HAMi</li>
<li class=""><strong>时间</strong>：9 月 8 日 11:14-11:19</li>
<li class=""><strong>地点</strong>：5B + C</li>
<li class=""><strong>讲者</strong>：王纪飞（HAMi Approver，密瓜智能）</li>
</ul>
<p>在 Kubernetes 中使用 NVIDIA MIG，通常需要预先静态切分：运维必须在工作负载到来之前决定好整张卡的分区布局。切少了浪费，切多了碎片化，等负载真正到来时，布局往往已经不对了。</p>
<p>这场 5 分钟的闪电演讲提出一条调度驱动的路线：HAMi 将调度器与 device plugin 集成，让 GPU 分区跟随实时调度决策动态调整：先调度、后切分，而不是先切分、再调度。</p>
<blockquote>
<p>GPU partitioning should follow scheduling, not precede it.（GPU 分区应该跟随调度，而不是先于调度。）</p>
</blockquote>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="专场分享合合信息如何支撑数十亿次文档扫描基于-hami-的千卡规模-gpu-虚拟化">专场分享：合合信息如何支撑数十亿次文档扫描，基于 HAMi 的千卡规模 GPU 虚拟化<a href="https://project-hami.io/zh/blog/hami-at-kubecon-china-2026#%E4%B8%93%E5%9C%BA%E5%88%86%E4%BA%AB%E5%90%88%E5%90%88%E4%BF%A1%E6%81%AF%E5%A6%82%E4%BD%95%E6%94%AF%E6%92%91%E6%95%B0%E5%8D%81%E4%BA%BF%E6%AC%A1%E6%96%87%E6%A1%A3%E6%89%AB%E6%8F%8F%E5%9F%BA%E4%BA%8E-hami-%E7%9A%84%E5%8D%83%E5%8D%A1%E8%A7%84%E6%A8%A1-gpu-%E8%99%9A%E6%8B%9F%E5%8C%96" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class=""><strong>原题</strong>：How Intsig Serves Billions of Document Scans: GPU Virtualization at Scale with HAMi</li>
<li class=""><strong>时间</strong>：9 月 8 日 14:30-15:00</li>
<li class=""><strong>地点</strong>：Grand Ballroom II + III</li>
<li class=""><strong>讲者</strong>：李孟轩（密瓜智能联合创始人兼 CTO）、Walter Duan（合合信息 Intsig）</li>
</ul>
<p>这是一堂千卡规模的生产实践课。<a href="https://www.intsig.com/" target="_blank" rel="noopener noreferrer" class="">合合信息</a>（CamScanner 扫描全能王，全球 3 亿 + 下载）的 GPU 场景非常极致：单一 OCR 负载、极高并发、约 1000 张 GPU 卡，瓶颈不在调度放置，而在排队时间。</p>
<p>两位讲者将分享如何从腾讯 QGPU 迁移到 HAMi，一站式获得虚拟化、调度与监控能力，实现排队时间下降、利用率上升。演讲中还将集中披露 HAMi 的生产案例数据：</p>
<ul>
<li class=""><strong>顺丰</strong>：GPU 从 1400 张精简到 1000 张，业务不受影响</li>
<li class=""><strong>招商银行</strong>：10000+ GPU，利用率 20% → 80%</li>
<li class=""><strong>蔚来</strong>：CI 效率提升 10 倍</li>
<li class=""><strong>工商银行</strong>：GPU 利用率 20% → 70%</li>
</ul>
<p>分享不止有“用得好的模式”，也有“烧钱的反模式”，例如 GPU 切分低于 1/6 时反而适得其反。结尾还将现场演示合合信息开源的 AI 终端 Chaterm：用自然语言直接操作 GPU 集群。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="hami-项目展台t-1">HAMi 项目展台：T-1<a href="https://project-hami.io/zh/blog/hami-at-kubecon-china-2026#hami-%E9%A1%B9%E7%9B%AE%E5%B1%95%E5%8F%B0t-1" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class=""><strong>展位</strong>：T-1（Grand Ballroom I）</li>
<li class=""><strong>时间</strong>：9 月 8 日 10:30-19:00</li>
</ul>
<p>HAMi 维护者将在展台值守。无论你正在做 GPU 共享、为算力利用率发愁，还是想了解多租户 GPU 管理，都欢迎来聊：</p>
<ul>
<li class="">GPU 虚拟化与共享：显存、算力的切分与隔离</li>
<li class="">AI 负载调度与利用率优化</li>
<li class="">MIG、vGPU 等多种切分方案的选型</li>
<li class="">与 <a href="https://volcano.sh/" target="_blank" rel="noopener noreferrer" class="">Volcano</a>、<a href="https://kueue.sigs.k8s.io/" target="_blank" rel="noopener noreferrer" class="">Kueue</a>、<a href="https://github.com/NVIDIA/KAI-Scheduler" target="_blank" rel="noopener noreferrer" class="">KAI Scheduler</a>、<a href="https://github.com/vllm-project/vllm" target="_blank" rel="noopener noreferrer" class="">vLLM</a> 等生态项目的集成</li>
</ul>
<p>现场备有社区贴纸和周边，扫码即可加入 HAMi 社区群、为项目点亮 Star。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="会前一站hami-meetup-上海站">会前一站：HAMi Meetup 上海站<a href="https://project-hami.io/zh/blog/hami-at-kubecon-china-2026#%E4%BC%9A%E5%89%8D%E4%B8%80%E7%AB%99hami-meetup-%E4%B8%8A%E6%B5%B7%E7%AB%99" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>如果你提前到上海：9 月 6 日（周日）下午，密瓜智能（Dynamia）和 HAMi 社区将在上海五角场联合主办 <strong>“不卷算力，卷效率”HAMi Meetup 上海站 · Incubating 特别活动</strong>。这是 HAMi 晋级孵化项目后的首场线下特别活动，涵盖主题演讲、社区圆桌、Incubating 里程碑环节与 Community Night。<a href="https://www.huodongxing.com/event/2874911381700" target="_blank" rel="noopener noreferrer" class="">点此报名</a>。</p>
<p>9 月 8 日，上海，HAMi 展台 T-1，我们不见不散。</p>]]></content>
        <author>
            <name>HAMi 社区</name>
        </author>
        <category label="KubeCon" term="KubeCon"/>
        <category label="GPU" term="GPU"/>
        <category label="Kubernetes" term="Kubernetes"/>
        <category label="AI" term="AI"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[HAMi v2.10.0 发布：Flexible MIG、可组合调度策略与更广阔的加速器生态]]></title>
        <id>https://project-hami.io/zh/blog/hami-v2-10-0-release</id>
        <link href="https://project-hami.io/zh/blog/hami-v2-10-0-release"/>
        <updated>2026-08-21T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[HAMi v2.10.0 正式发布。本次发布带来了动态 Flexible MIG、全新的 mutex 调度策略、NUMA 排序修复与可组合调度策略、PodGroup（gang-scheduling）支持、AMD MI300X 与壁仞设备支持、更深入的昇腾管理（vNPU 与 HAMi-core 异构模式及监控），以及 KAI Scheduler + HAMi-core 集成。]]></summary>
        <content type="html"><![CDATA[<p>HAMi 社区正式发布 <strong>HAMi v2.10.0</strong>。本次发布在三个方向上取得进展：<strong>更灵活的调度策略、更广泛的异构加速器覆盖，以及更加丰富的调度器生态</strong>。</p>
<p>v2.10.0 引入了动态 <strong>Flexible MIG</strong>、全新的 <strong>mutex</strong> 调度策略、社区期待已久的 <strong>NUMA 排序修复</strong>、<strong>可组合调度策略</strong>、<strong>PodGroup（gang-scheduling）</strong> 支持，以及更准确的 <strong>init 容器</strong> 资源计量。在设备侧，新增了 <strong>AMD MI300X</strong> 与<strong>壁仞</strong>支持、让基于模板的 vNPU 与 HAMi-core 节点在同一集群共存的<strong>昇腾异构管理</strong>能力，以及 <strong>vNPU HAMi-core 监控</strong>。同时通过全新的 KAI Resource Isolator 伴随项目，首次实现了 <strong>KAI Scheduler + HAMi-core</strong> 集成。</p>
<p>本文将对 v2.10.0 的主要更新进行详细说明。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="调度更灵活更可组合">调度：更灵活、更可组合<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#%E8%B0%83%E5%BA%A6%E6%9B%B4%E7%81%B5%E6%B4%BB%E6%9B%B4%E5%8F%AF%E7%BB%84%E5%90%88" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="flexible-mig无需驱逐节点的动态-mig">Flexible MIG：无需驱逐节点的动态 MIG<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#flexible-mig%E6%97%A0%E9%9C%80%E9%A9%B1%E9%80%90%E8%8A%82%E7%82%B9%E7%9A%84%E5%8A%A8%E6%80%81-mig" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>过去在 HAMi 中使用 NVIDIA MIG（Multi-Instance GPU），往往依赖静态 MIG 几何与 <code>nvidia-mig-parted</code> 等工具，在变更 profile 前通常需要先 drain 或 cordon 节点。v2.10.0 用 <strong>Flexible MIG</strong> 取代了这一流程，实现按需动态创建与销毁 MIG 实例（<a href="https://github.com/Project-HAMi/HAMi/pull/2378" target="_blank" rel="noopener noreferrer" class="">#2378</a>，<a href="https://github.com/FouoF" target="_blank" rel="noopener noreferrer" class="">@FouoF</a>）。</p>
<p>主要亮点：</p>
<ul>
<li class=""><strong>无需驱逐</strong>：MIG 实例随工作负载的进出动态创建与销毁，无需 cordon 节点。</li>
<li class=""><strong>NVML 动态发现</strong>：通过 NVML 动态发现 profile，结合拓扑感知的资源预留与 profile 白名单来约束可切分的范围。</li>
<li class=""><strong>重启后状态保留</strong>：MIG 预留与运行时身份（原生 GI/CI 状态）可在 device-plugin 恢复与节点重启后保留，并通过 Pod 注解记录。</li>
<li class=""><strong>更安全的启动</strong>：device plugin 会避免重置已有活跃工作负载的 GPU。</li>
<li class=""><strong>MIG 指标现代化</strong>：DCGM 风格的指标暴露 MIG UUID、profile、实例 ID 与位置坐标。</li>
</ul>
<blockquote>
<p><strong>已知限制</strong>：CDI 模式暂不支持与 MIG 同时使用，多设备 MIG 场景仍需进一步验证。建议在生产环境部署前，针对你的实际拓扑进行测试。</p>
</blockquote>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="mutex-调度策略与-numa-排序修复">Mutex 调度策略与 NUMA 排序修复<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#mutex-%E8%B0%83%E5%BA%A6%E7%AD%96%E7%95%A5%E4%B8%8E-numa-%E6%8E%92%E5%BA%8F%E4%BF%AE%E5%A4%8D" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>v2.10.0 新增了 <strong><code>mutex</code></strong> GPU 调度策略（<a href="https://github.com/Project-HAMi/HAMi/pull/2011" target="_blank" rel="noopener noreferrer" class="">#2011</a>，<a href="https://github.com/mesutoezdil" target="_blank" rel="noopener noreferrer" class="">@mesutoezdil</a>）。带有 <code>hami.io/gpu-scheduler-policy: mutex</code> 注解的 Pod 会被调度到<strong>当前没有任何用户使用的 GPU</strong> 上，从而为该工作负载保证独占的设备访问，非常适合对延迟敏感的推理，或不能共享同一颗 die 的工作负载。</p>
<p>同一改动还修复了一个长期存在、影响 <code>binpack</code> 与 <code>spread</code> 的排序 bug。此前调度器以 <strong>NUMA 节点作为主排序键</strong>，导致无论实际负载如何，工作负载都被固定到某个 NUMA 节点（<a href="https://github.com/Project-HAMi/HAMi/issues/1806" target="_blank" rel="noopener noreferrer" class="">#1806</a>）。本次发布后，<strong>设备利用率 Score 成为主排序键，NUMA 仅作为 tiebreaker</strong>，因此 <code>binpack</code> 与 <code>spread</code> 在多 NUMA 拓扑下的行为恢复正常（<a href="https://github.com/Project-HAMi/HAMi/pull/2011" target="_blank" rel="noopener noreferrer" class="">#2011</a>）。</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 请求独占分配的 GPU</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">metadata</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">annotations</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">hami.io/gpu-scheduler-policy</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"mutex"</span><br></div></code></pre></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="可组合调度策略">可组合调度策略<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#%E5%8F%AF%E7%BB%84%E5%90%88%E8%B0%83%E5%BA%A6%E7%AD%96%E7%95%A5" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>单个策略各有用处，但真实集群往往需要多种行为叠加：既要紧凑装箱，又要保持 NUMA 亲和，还要留几张卡独占。v2.10.0 让 <code>hami.io/gpu-scheduler-policy</code> 接受<strong>有序的、逗号分隔的列表</strong>，使过滤型与排序型策略可以组合（<a href="https://github.com/Project-HAMi/HAMi/pull/2621" target="_blank" rel="noopener noreferrer" class="">#2621</a>，<a href="https://github.com/mesutoezdil" target="_blank" rel="noopener noreferrer" class="">@mesutoezdil</a>，关闭 <a href="https://github.com/Project-HAMi/HAMi/issues/2010" target="_blank" rel="noopener noreferrer" class="">#2010</a>）：</p>
<ul>
<li class=""><code>binpack</code>、<code>spread</code>、<code>numa</code> 作为<strong>排序键，按书写顺序生效</strong>：<code>binpack,numa</code> 先按 binpack 排序，再以 NUMA 作为 tiebreaker。</li>
<li class=""><code>mutex</code>（以及 NVIDIA 上的 <code>topology-aware</code>）作为<strong>过滤器</strong>，在排序前先筛掉不合格的候选设备。</li>
<li class="">比较结果相同的候选按设备索引确定性地排序；仅含过滤器的组合回退为 <code>spread</code>。</li>
<li class="">逗号两侧的空白会被忽略，<strong>单个策略值的行为与之前完全一致</strong>。</li>
</ul>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 仅独占 GPU，再紧凑装箱，NUMA 作为 tiebreaker</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">metadata</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">annotations</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">hami.io/gpu-scheduler-policy</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"mutex,binpack,numa"</span><br></div></code></pre></div></div>
<p>至此，v2.10 roadmap 中跟踪的调度策略组合工作宣告完成：各策略（<code>mutex</code>、<code>numa</code>、<code>topology-aware</code>、<code>binpack</code>、<code>spread</code>）既可以单独使用，也可以按任意顺序组合成链。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="podgroupgang-scheduling支持">PodGroup（gang-scheduling）支持<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#podgroupgang-scheduling%E6%94%AF%E6%8C%81" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>针对分布式训练等"要么全调度、要么不调度"的工作负载，v2.10.0 新增了 <strong>PodGroup 支持</strong>，使同一 PodGroup 的成员在节点锁竞争时能干净地完成 bind，而不会互相挤掉（<a href="https://github.com/Project-HAMi/HAMi/pull/2066" target="_blank" rel="noopener noreferrer" class="">#2066</a>，<a href="https://github.com/lin121291" target="_blank" rel="noopener noreferrer" class="">@lin121291</a>，issue <a href="https://github.com/Project-HAMi/HAMi/issues/1832" target="_blank" rel="noopener noreferrer" class="">#1832</a>）。</p>
<p>此前，当同一 PodGroup 的多个成员并发 bind 到同一节点时，它们会争抢 <code>hami.io/mutex.lock</code> 注解；失败者会立即失败并退回到 kube-scheduler 的退避逻辑。v2.10.0 为 PodGroup 成员（通过 <code>scheduling.x-k8s.io/pod-group</code> 标签识别）在 <code>Bind</code> 中新增了<strong>重试循环</strong>，并引入新的 <code>--node-lock-retry-timeout</code> 参数（默认 28 秒，刻意低于 extender 30 秒的超时）。非 PodGroup 的 Pod 行为保持不变。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="更准确的-init-容器资源计量">更准确的 init 容器资源计量<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#%E6%9B%B4%E5%87%86%E7%A1%AE%E7%9A%84-init-%E5%AE%B9%E5%99%A8%E8%B5%84%E6%BA%90%E8%AE%A1%E9%87%8F" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>使用 init 容器下载权重或准备数据的 Pod 此前会被多算：调度器将<em>所有</em>容器请求相加，把 init 容器当作并行运行来处理。v2.10.0 让 HAMi 对齐了标准 Kubernetes 语义：有效请求变为 <code>max(Σ 应用容器, max(init 容器))</code>（<a href="https://github.com/Project-HAMi/HAMi/pull/1773" target="_blank" rel="noopener noreferrer" class="">#1773</a>，<a href="https://github.com/maishivamhoo123" target="_blank" rel="noopener noreferrer" class="">@maishivamhoo123</a>）。这消除了依赖 init 容器的 Pod 出现的误报"节点已满"失败与错误的配额拒绝。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="更广泛的异构加速器支持">更广泛的异构加速器支持<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#%E6%9B%B4%E5%B9%BF%E6%B3%9B%E7%9A%84%E5%BC%82%E6%9E%84%E5%8A%A0%E9%80%9F%E5%99%A8%E6%94%AF%E6%8C%81" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="amd-instinct-mi300x-vgpu">AMD Instinct MI300X vGPU<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#amd-instinct-mi300x-vgpu" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>v2.10.0 新增了对 <strong>AMD Instinct 加速器的软件 vGPU 支持</strong>，已在 <strong>MI300X</strong> 上基于 ROCm 7.0.2 验证（<a href="https://github.com/Project-HAMi/HAMi/pull/2290" target="_blank" rel="noopener noreferrer" class="">#2290</a>，<a href="https://github.com/FouoF" target="_blank" rel="noopener noreferrer" class="">@FouoF</a>、<a href="https://github.com/kenji-mido" target="_blank" rel="noopener noreferrer" class="">@kenji-mido</a>）。</p>
<p>该方案不依赖硬件 SR-IOV，而是通过 glibc <code>LD_AUDIT</code> 注入用户态 hook（<code>libamvgpu.so</code>），并借助 <code>HIP_DEVICE_MEMORY_LIMIT</code> 实施限制。这为显存与算力都提供了硬隔离：</p>
<table><thead><tr><th style="text-align:left">维度</th><th style="text-align:left">请求方式</th><th style="text-align:left">行为</th></tr></thead><tbody><tr><td style="text-align:left">显存隔离</td><td style="text-align:left"><code>amd.com/gpumem</code>（MiB）</td><td style="text-align:left">单 GPU 硬限制；使用量不会超过分配值</td></tr><tr><td style="text-align:left">算力隔离</td><td style="text-align:left"><code>amd.com/gpucores</code>（0-100，CU%）</td><td style="text-align:left">限制计算单元百分比（如 <code>25</code> ≈ 304 CU 设备上的 76 个 CU）</td></tr><tr><td style="text-align:left">设备数量</td><td style="text-align:left"><code>amd.com/gpu</code></td><td style="text-align:left">GPU 数量</td></tr></tbody></table>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">limits</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">amd.com/gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"1"</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 1 张 GPU</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">amd.com/gpumem</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"49152"</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 48 GiB 显存</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">amd.com/gpucores</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"30"</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 30% 计算单元</span><br></div></code></pre></div></div>
<p>AMD 支持以独立的 <a href="https://github.com/Project-Hami/amd-device-plugin" target="_blank" rel="noopener noreferrer" class=""><code>amd-device-plugin</code></a> 组件（<code>0.0.1</code>）提供，通过 <code>postStart</code> hook 安装 <code>libamvgpu.so</code>。<strong>请勿将其与上游 ROCm <code>k8s-device-plugin</code> 混用。</strong></p>
<div class="theme-admonition theme-admonition-warning admonition_xJq3 alert alert--warning"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 16 16"><path fill-rule="evenodd" d="M8.893 1.5c-.183-.31-.52-.5-.887-.5s-.703.19-.886.5L.138 13.499a.98.98 0 0 0 0 1.001c.193.31.53.501.886.501h13.964c.367 0 .704-.19.877-.5a1.03 1.03 0 0 0 .01-1.002L8.893 1.5zm.133 11.497H6.987v-2.003h2.039v2.003zm0-3.004H6.987V5.987h2.039v4.006z"></path></svg></span>要求与限制</div><div class="admonitionContent_BuS1"><ul>
<li class="">工作负载镜像必须基于 <strong>glibc 且 GLIBC ≥ 2.34</strong>（如较新的 <code>rocm/pytorch</code> 标签）。<strong>暂不支持 Alpine/musl、Ubuntu 20.04 与 RHEL 8</strong>（<a href="https://github.com/Project-HAMi/HAMi/issues/2265" target="_blank" rel="noopener noreferrer" class="">#2265</a>）。</li>
<li class="">暂不支持基于 <strong>RDNA/WGP</strong> 的设备。</li>
<li class="">受限于硬件，单 Pod 多 GPU 场景尚未验证。</li>
<li class="">若使用 AMD GPU Operator 安装驱动，请先关闭其内置的 device plugin。</li>
</ul></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="壁仞biren设备支持">壁仞（Biren）设备支持<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#%E5%A3%81%E4%BB%9Ebiren%E8%AE%BE%E5%A4%87%E6%94%AF%E6%8C%81" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>v2.10.0 新增了对<strong>壁仞</strong>加速器的管理支持（已验证型号：<code>Biren166M</code>），由 <a href="https://github.com/DSFans2014" target="_blank" rel="noopener noreferrer" class="">@DSFans2014</a> 贡献（<a href="https://github.com/Project-HAMi/HAMi/pull/1711" target="_blank" rel="noopener noreferrer" class="">#1711</a>）。提供两种分配模式：</p>
<ul>
<li class=""><strong>整卡模式（Full-card）</strong>：每个 Pod 独占一整张 GPU。</li>
<li class=""><strong>SVI 切分</strong>：将一张卡固定切分为 <strong>2 或 4 个分区</strong>，支持通过注解选择/排除设备 UUID。</li>
</ul>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 节点需要先打标签：kubectl label node &lt;node-name&gt; biren=on</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">limits</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">birentech.com/gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"1"</span><br></div></code></pre></div></div>
<blockquote>
<p><strong>注意</strong>：与 NVIDIA/AMD 不同，壁仞<strong>不能</strong>按 Pod 指定显存或算力大小，你得到的是整卡或一个 SVI 分区（2 个分区或 4 个分区）。<code>biren-device-plugin</code> DaemonSet 部署在 <code>biren-gpu</code> 命名空间。</p>
</blockquote>
<p>AMD 与壁仞的加入，使 HAMi 已覆盖更广泛的加速器，包括 NVIDIA、华为昇腾、寒武纪、海光 DCU、壁仞、燧原、沐曦、昆仑芯、天数智芯、AWS Neuron 与瀚博半导体。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="更灵活的昇腾管理">更灵活的昇腾管理<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#%E6%9B%B4%E7%81%B5%E6%B4%BB%E7%9A%84%E6%98%87%E8%85%BE%E7%AE%A1%E7%90%86" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="昇腾异构模式vnpu-与-hami-core-同集群共存">昇腾异构模式：vNPU 与 HAMi-core 同集群共存<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#%E6%98%87%E8%85%BE%E5%BC%82%E6%9E%84%E6%A8%A1%E5%BC%8Fvnpu-%E4%B8%8E-hami-core-%E5%90%8C%E9%9B%86%E7%BE%A4%E5%85%B1%E5%AD%98" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>HAMi v2.9.0 为昇腾引入了 HAMi-core 模式（基于软件的显存与算力切分）。但此前仍有一个实际限制：Pod 必须显式选择某种模式，集群运维人员实际上要维护两套工作负载配置。</p>
<p>v2.10.0 让昇腾工作负载变为<strong>模式无关（mode-agnostic）</strong>。不带 <code>huawei.com/vnpu-mode</code> 注解的 Pod 会跟随其落地节点所支持的模式在同一集群内自动适配：在模板节点上使用<strong>基于模板的硬切分（vNPU）</strong>，在 HAMi-core 节点上使用 <strong>HAMi-core 软切分</strong>（<a href="https://github.com/Project-HAMi/HAMi/pull/2035" target="_blank" rel="noopener noreferrer" class="">HAMi #2035</a> 与 <a href="https://github.com/Project-HAMi/ascend-device-plugin/pull/106" target="_blank" rel="noopener noreferrer" class="">ascend-device-plugin #106</a>，<a href="https://github.com/ouyangluwei163" target="_blank" rel="noopener noreferrer" class="">@ouyangluwei163</a>）。带有显式 <code>huawei.com/vnpu-mode: hami-core</code> 注解的 Pod 仍严格绑定到该模式。调度器仅在节点确实不支持 hami-core 时，才会拒绝 hami-core 请求。</p>
<p>这降低了运行混合昇腾机群的复杂度：不再需要为两种切分方式维护两份工作负载清单。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="vnpu-hami-core-监控">vNPU HAMi-core 监控<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#vnpu-hami-core-%E7%9B%91%E6%8E%A7" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>软切分的昇腾资源现在不仅"可分配"，更<strong>可观测</strong>。v2.10.0 为 vNPU HAMi-core 模式开启了内嵌的 Prometheus 指标服务，提供容器级与设备级的可见性：</p>
<ul>
<li class=""><strong>9395 端口的 metrics 服务</strong>，仅在启用 <code>hami-vnpu-core</code> 模式时启动（<a href="https://github.com/Project-HAMi/ascend-device-plugin/pull/93" target="_blank" rel="noopener noreferrer" class="">ascend-device-plugin #93</a>，<a href="https://github.com/maverick123123" target="_blank" rel="noopener noreferrer" class="">@maverick123123</a>）。</li>
<li class=""><strong>容器级 AICore 利用率</strong>：<code>hami_container_device_utilization_ratio</code>（按设备 UUID 正确映射，而非回退到第一个设备）。</li>
<li class=""><strong>设备级显存</strong>：<code>hami_host_gpu_memory_used_bytes</code>（按容器用量聚合，确保共享场景下的准确性）。</li>
<li class="">基于 DCMI 的<strong>进程级 HBM 追踪</strong>，并提供容器级共享内存计量，同时支持多设备（TP&gt;1）推理（<a href="https://github.com/Project-HAMi/ascend-device-plugin/pull/87" target="_blank" rel="noopener noreferrer" class="">ascend-device-plugin #87</a>；<a href="https://github.com/Project-HAMi/hami-vnpu-core/pull/10" target="_blank" rel="noopener noreferrer" class="">hami-vnpu-core #10</a>）。</li>
<li class="">通过 ascend-device-plugin chart <strong>支持 Helm 部署</strong>的监控（<a href="https://github.com/Project-HAMi/ascend-device-plugin/pull/108" target="_blank" rel="noopener noreferrer" class="">#108</a>，<a href="https://github.com/DSFans2014" target="_blank" rel="noopener noreferrer" class="">@DSFans2014</a>）。</li>
</ul>
<p>这让软切分昇腾资源从"可分配"迈向"可观测、可运营"。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="调度器生态">调度器生态<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#%E8%B0%83%E5%BA%A6%E5%99%A8%E7%94%9F%E6%80%81" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="通过-kai-resource-isolator-实现-kai-scheduler--hami-core">通过 KAI Resource Isolator 实现 KAI Scheduler + HAMi-core<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#%E9%80%9A%E8%BF%87-kai-resource-isolator-%E5%AE%9E%E7%8E%B0-kai-scheduler--hami-core" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>v2.10.0 首次推出了一个集成方案：让 <strong>KAI Scheduler</strong> 负责 GPU 共享的调度决策，而 <strong>HAMi-core</strong> 负责容器内的运行时隔离，两者由全新的伴随项目 <a href="https://github.com/Project-HAMi/KAI-resource-isolator" target="_blank" rel="noopener noreferrer" class="">KAI Resource Isolator</a>（v1.1.0，<a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">@archlitchi</a>；监控由 <a href="https://github.com/dttung2905" target="_blank" rel="noopener noreferrer" class="">@dttung2905</a> 贡献）连接。</p>
<p>职责划分非常清晰：</p>
<ul>
<li class=""><strong>KAI Scheduler</strong> 决定<em>哪个</em> Pod 获得<em>多少</em> GPU 份额。</li>
<li class=""><strong>KAI Resource Isolator</strong> 让该决策<em>真正生效</em>：DaemonSet（<code>libsync</code>）将 HAMi-core 的 <code>libvgpu.so</code> 下发到 GPU 节点，<strong>mutating webhook</strong> 把该库注入到共享 Pod 中并修改 <code>/etc/ld.so.preload</code>，使 <code>libvgpu.so</code> 拦截 CUDA 内存分配调用，强制执行 KAI Scheduler 通过 <code>CUDA_DEVICE_MEMORY_LIMIT</code> 设定的显存硬限制。</li>
<li class="">该集成<strong>复用 HAMi 的监控指标</strong>，并与现有的 HAMi Grafana 仪表盘兼容。</li>
</ul>
<p>需要 KAI Scheduler <strong>≥ 0.17.0</strong>，并设置 <code>global.gpuSharing=true</code> 与 <code>binder.plugins.hamicore.enabled=true</code>。</p>
<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"></path></svg></span>持续完善中</div><div class="admonitionContent_BuS1"><p>KAI + HAMi-core 的架构已经就位，isolator 也已发布，但有两个加固 PR 仍在进行中，在生产级硬隔离场景使用前值得跟踪：基于 fraction 的显存强制执行（<a href="https://github.com/Project-HAMi/KAI-resource-isolator/pull/6" target="_blank" rel="noopener noreferrer" class="">#6</a>）与非 root 容器目录权限（<a href="https://github.com/Project-HAMi/KAI-resource-isolator/pull/22" target="_blank" rel="noopener noreferrer" class="">#22</a>）。建议在你的 KAI GPU 共享 Pod 中验证显存限制确实生效。HAMi 侧相关的 CRD 工作（<a href="https://github.com/Project-HAMi/HAMi/pull/2014" target="_blank" rel="noopener noreferrer" class="">#2014</a>）也仍处于 draft 状态。可参阅我们此前的深度文章<a class="" href="https://project-hami.io/zh/blog/kai-scheduler-hami-gpu-memory-hard-isolation">《KAI Scheduler + HAMi：GPU 显存硬隔离》</a>。</p></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="volcano-支持昇腾软切分-vnpu">Volcano 支持昇腾软切分 vNPU<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#volcano-%E6%94%AF%E6%8C%81%E6%98%87%E8%85%BE%E8%BD%AF%E5%88%87%E5%88%86-vnpu" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>Volcano 现在可以结合 Ascend device plugin 与 <code>hami-vnpu-core</code>，调度并运行昇腾软切分工作负载，把 Volcano 的批量调度能力与 HAMi 的运行时隔离能力结合起来。Volcano 通过其 <code>deviceshare</code> 插件（<code>AscendHAMiVNPUEnable: true</code>）暴露昇腾 vNPU，支持包括 910A、910B2、910B3、310P 在内的异构昇腾集群。</p>
<p>需要留意的前提条件：需要 <strong>Volcano ≥ 1.14</strong>，<strong><code>hami-core</code> 软切分需要 ≥ 1.16</strong>，且软切分<strong>仅支持 ARM 平台</strong>。详见 <a href="https://github.com/Project-HAMi/ascend-device-plugin/blob/main/docs/volcano.md" target="_blank" rel="noopener noreferrer" class="">ascend-device-plugin Volcano 指南</a>。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="chart构建与运维改进">Chart、构建与运维改进<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#chart%E6%9E%84%E5%BB%BA%E4%B8%8E%E8%BF%90%E7%BB%B4%E6%94%B9%E8%BF%9B" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class=""><strong>DRA 迁移到独立 chart</strong>（<a href="https://github.com/Project-HAMi/HAMi/pull/2038" target="_blank" rel="noopener noreferrer" class="">#2038</a>，<a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">@archlitchi</a>）：DRA 组件从主 HAMi Helm chart 中移除。DRA 现在拥有<a href="https://github.com/Project-HAMi/HAMi-dra" target="_blank" rel="noopener noreferrer" class="">独立的 chart/仓库</a>，不使用 DRA 的集群不再需要拉取这些组件。</li>
<li class=""><strong>握手注解优化</strong>（<a href="https://github.com/Project-HAMi/HAMi/pull/2052" target="_blank" rel="noopener noreferrer" class="">#2052</a>，<a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">@archlitchi</a>）：节点清理时现在会完整移除握手注解键，而不是写入带时间戳的 <code>Deleted_*</code> 标记，使健康/重置上报保持一致。</li>
<li class=""><strong>ubi8 编译镜像</strong>（<a href="https://github.com/Project-HAMi/HAMi/pull/1958" target="_blank" rel="noopener noreferrer" class="">#1958</a>，<a href="https://github.com/spencercjh" target="_blank" rel="noopener noreferrer" class="">@spencercjh</a>）：HAMi 与 HAMi-core 的构建阶段迁移到 <code>nvidia/cuda:13.3.0-cudnn-devel-ubi8</code>，扩大了 GLIBC 兼容范围，使构建产物能在更广泛的目标发行版上运行。</li>
<li class=""><strong>mock-device-plugin NPU 模板</strong>（<a href="https://github.com/Project-HAMi/mock-device-plugin/pull/18" target="_blank" rel="noopener noreferrer" class="">mock-device-plugin #18</a>，<a href="https://github.com/Wangmin362" target="_blank" rel="noopener noreferrer" class="">@Wangmin362</a>）：mock 插件现在同时支持新版嵌套格式与旧版扁平 <code>vnpus</code> 配置，并注册了昇腾 AI-core 与海光 DCU cores 资源，支持在无硬件环境下测试 NPU vNPU 调度。</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="贡献者">贡献者<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#%E8%B4%A1%E7%8C%AE%E8%80%85" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>v2.10.0 是 HAMi 主仓库与更广泛的 Project-HAMi 组织（ascend-device-plugin、KAI-resource-isolator、mock-device-plugin、hami-vnpu-core、amd-device-plugin）共同贡献的成果。本周期代表性功能负责人包括：</p>
<ul>
<li class=""><a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">@archlitchi</a>：发布协调、KAI Resource Isolator、DRA chart 拆分、握手优化</li>
<li class=""><a href="https://github.com/mesutoezdil" target="_blank" rel="noopener noreferrer" class="">@mesutoezdil</a>：mutex 调度策略、NUMA 排序修复、可组合策略链</li>
<li class=""><a href="https://github.com/maverick123123" target="_blank" rel="noopener noreferrer" class="">@maverick123123</a>：昇腾 vNPU HAMi-core 监控</li>
<li class=""><a href="https://github.com/ouyangluwei163" target="_blank" rel="noopener noreferrer" class="">@ouyangluwei163</a>：昇腾异构模式（vNPU + HAMi-core）</li>
<li class=""><a href="https://github.com/FouoF" target="_blank" rel="noopener noreferrer" class="">@FouoF</a>、<a href="https://github.com/kenji-mido" target="_blank" rel="noopener noreferrer" class="">@kenji-mido</a>：AMD MI300X vGPU、Flexible MIG</li>
<li class=""><a href="https://github.com/DSFans2014" target="_blank" rel="noopener noreferrer" class="">@DSFans2014</a>：壁仞设备支持、vNPU 监控 Helm chart</li>
<li class=""><a href="https://github.com/lin121291" target="_blank" rel="noopener noreferrer" class="">@lin121291</a>：PodGroup gang-scheduling 支持</li>
<li class=""><a href="https://github.com/maishivamhoo123" target="_blank" rel="noopener noreferrer" class="">@maishivamhoo123</a>：init 容器资源计量</li>
<li class=""><a href="https://github.com/Wangmin362" target="_blank" rel="noopener noreferrer" class="">@Wangmin362</a>：mock-device-plugin NPU 模板</li>
<li class=""><a href="https://github.com/spencercjh" target="_blank" rel="noopener noreferrer" class="">@spencercjh</a>：ubi8 编译镜像</li>
<li class=""><a href="https://github.com/dttung2905" target="_blank" rel="noopener noreferrer" class="">@dttung2905</a>：KAI vGPU 监控与非 root 工作负载修复</li>
</ul>
<p>感谢每一位贡献者的付出，也感谢每一位报告问题、测试候选版本、分享生产反馈的用户。完整贡献者名单可见 <a href="https://github.com/Project-HAMi/HAMi/releases/tag/v2.10.0" target="_blank" rel="noopener noreferrer" class="">v2.10.0 Release Notes</a>。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="升级指南">升级指南<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#%E5%8D%87%E7%BA%A7%E6%8C%87%E5%8D%97" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>通过 Helm 升级至 v2.10.0：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm repo </span><span class="token function" style="color:rgb(130, 170, 255)">add</span><span class="token plain"> hami-charts https://project-hami.github.io/HAMi/</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm repo update</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm upgrade hami hami-charts/hami </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> kube-system</span><br></div></code></pre></div></div>
<p>完整安装文档请参考：<a class="" href="https://project-hami.io/zh/docs/installation/online-installation">https://project-hami.io/zh/docs/installation/online-installation</a></p>
<div class="theme-admonition theme-admonition-warning admonition_xJq3 alert alert--warning"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 16 16"><path fill-rule="evenodd" d="M8.893 1.5c-.183-.31-.52-.5-.887-.5s-.703.19-.886.5L.138 13.499a.98.98 0 0 0 0 1.001c.193.31.53.501.886.501h13.964c.367 0 .704-.19.877-.5a1.03 1.03 0 0 0 .01-1.002L8.893 1.5zm.133 11.497H6.987v-2.003h2.039v2.003zm0-3.004H6.987V5.987h2.039v4.006z"></path></svg></span>升级注意事项</div><div class="admonitionContent_BuS1"><ul>
<li class=""><strong>DRA 用户</strong>：DRA 不再随主 chart 安装。若依赖 HAMi-DRA，请按 <a href="https://github.com/Project-HAMi/HAMi-dra" target="_blank" rel="noopener noreferrer" class="">HAMi-dra</a> chart 进行安装。</li>
<li class=""><strong>AMD 用户</strong>：请部署独立的 <a href="https://github.com/Project-Hami/amd-device-plugin" target="_blank" rel="noopener noreferrer" class=""><code>amd-device-plugin</code></a>，并确保工作负载镜像满足 GLIBC 2.34 要求。</li>
<li class=""><strong>昇腾用户</strong>：vNPU HAMi-core 监控需启用 <code>hami-core</code> 模式；请核实 Volcano 版本（软切分需 ≥ 1.16，仅支持 ARM）。</li>
<li class="">建议在升级前于测试环境验证兼容性。</li>
</ul></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="总结">总结<a href="https://project-hami.io/zh/blog/hami-v2-10-0-release#%E6%80%BB%E7%BB%93" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>HAMi v2.10.0 是一次面向<strong>调度灵活性、加速器广度与生态深度</strong>的版本更新。借助 Flexible MIG、可组合调度策略（mutex、NUMA 排序修复与逗号分隔的策略链）、gang-scheduling 支持、AMD 与壁仞设备、更深入的昇腾管理与可观测性，以及 KAI Scheduler + HAMi-core 集成，HAMi 持续拓展统一异构算力调度平台的能力边界。</p>
<p>我们诚挚欢迎更多开发者、用户和生态伙伴参与 HAMi 社区，共同推动 GPU 虚拟化与异构算力调度能力的演进。</p>
<hr>
<p><strong>相关链接：</strong></p>
<ul>
<li class="">GitHub Release：<a href="https://github.com/Project-HAMi/HAMi/releases/tag/v2.10.0" target="_blank" rel="noopener noreferrer" class="">https://github.com/Project-HAMi/HAMi/releases/tag/v2.10.0</a></li>
<li class="">KAI Resource Isolator：<a href="https://github.com/Project-HAMi/KAI-resource-isolator" target="_blank" rel="noopener noreferrer" class="">https://github.com/Project-HAMi/KAI-resource-isolator</a></li>
<li class="">Ascend Device Plugin：<a href="https://github.com/Project-HAMi/ascend-device-plugin" target="_blank" rel="noopener noreferrer" class="">https://github.com/Project-HAMi/ascend-device-plugin</a></li>
<li class="">AMD Device Plugin：<a href="https://github.com/Project-Hami/amd-device-plugin" target="_blank" rel="noopener noreferrer" class="">https://github.com/Project-Hami/amd-device-plugin</a></li>
<li class="">项目文档：<a class="" href="https://project-hami.io/zh/docs">https://project-hami.io/zh/docs/</a></li>
<li class="">社区 Discord（推荐）：<a href="https://discord.gg/Amhy7XmbNq" target="_blank" rel="noopener noreferrer" class="">https://discord.gg/Amhy7XmbNq</a></li>
<li class="">社区 CNCF Slack：<a href="https://cloud-native.slack.com/archives/C07T10BU4R2" target="_blank" rel="noopener noreferrer" class="">https://cloud-native.slack.com/archives/C07T10BU4R2</a></li>
</ul>]]></content>
        <author>
            <name>HAMi 社区</name>
        </author>
        <category label="Release" term="Release"/>
        <category label="GPU" term="GPU"/>
        <category label="Kubernetes" term="Kubernetes"/>
        <category label="调度" term="调度"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Kubernetes DRA 会取代 HAMi 吗？]]></title>
        <id>https://project-hami.io/zh/blog/does-kubernetes-dra-replace-hami</id>
        <link href="https://project-hami.io/zh/blog/does-kubernetes-dra-replace-hami"/>
        <updated>2026-08-20T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[DRA 吸收了 GPU 共享中请求与调度的那一半，HAMi 保留了运行时强制执行的那一半。本文解释两者如何配合、如今跑通 DRA 栈需要什么，以及 2026 年年中该如何选择。]]></summary>
        <content type="html"><![CDATA[<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"></path></svg></span>备注</div><div class="admonitionContent_BuS1"><p>本文译自 <a href="https://www.cncf.io/blog/2026/08/07/does-kubernetes-dra-replace-hami/" target="_blank" rel="noopener noreferrer" class="">CNCF 博客</a>（2026 年 8 月 7 日），原文作者系 HAMi 项目贡献者。HAMi 已于 2026 年 7 月 15 日被 CNCF 技术监督委员会（TOC）接收为孵化项目。</p></div></div>
<p>想在一个 Kubernetes 上共享 GPU 的项目，长期以来只能「绕开」API 去工作，而不是「顺着」API 来工作。设备插件（device plugin）接口能做的事就是数设备，这就是它的全部词汇：<code>nvidia.com/gpu: 1</code>，意思是一整张卡，爱要不要。</p>
<p>HAMi 正是在这个词汇贫乏的基础上，构建了一整套流水线（变更型准入 Webhook、调度器扩展器、注解、容器内强制限制），去表达那些词汇表达不了的需求：「给这个 Pod 8000 MiB 显存和 10% 的算力，而且要把限制落到实处。」</p>
<p>后来，词汇变了。<strong>动态资源分配</strong>（Dynamic Resource Allocation，DRA） 在 Kubernetes v1.34 中正式发布（GA），并在 v1.35 起默认开启。借助其中的<strong>可消费容量</strong>（consumable capacity）特性，一个 Pod 现在可以越过任何注解，直接向调度器原生地申请一张设备显存的一个切片。</p>
<p>所以，HAMi 社区里反复被问到的问题是：DRA 会让 HAMi 过时吗？</p>
<p>简短的回答是：不会。但完整的回答取决于你指的是 HAMi 的哪项职责。其中一项，把碎片化请求编码成调度器看得懂的形式，恰恰是 DRA 要吸收掉的；另一项，在容器内以 CUDA 调用粒度强制执行这些配额，则是 DRA 从设计上就无意承担的工作。HAMi 的应对之策也顺势分成了两半：保留强制执行层，并在 DRA 之上用 3 个仓库重建编码层。</p>
<p>下面我们就把这两半拆开看清楚，再看如今跑通 DRA 这套栈需要什么。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="当-api-帮不上忙时gpu-共享是如何工作的">当 API 帮不上忙时，GPU 共享是如何工作的<a href="https://project-hami.io/zh/blog/does-kubernetes-dra-replace-hami#%E5%BD%93-api-%E5%B8%AE%E4%B8%8D%E4%B8%8A%E5%BF%99%E6%97%B6gpu-%E5%85%B1%E4%BA%AB%E6%98%AF%E5%A6%82%E4%BD%95%E5%B7%A5%E4%BD%9C%E7%9A%84" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>要看清 DRA 改变了什么，先得精确界定它取代的是哪套机器：HAMi 流水线里，哪些部件仅仅因为设备插件 API 太窄而存在。</p>
<p>一个 HAMi 用户会写 3 个扩展资源：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">limits</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">nvidia.com/gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 来自 1 张物理卡的设备 ID</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">nvidia.com/gpumem</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">8000</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 该卡 VRAM 的 MiB 数</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">nvidia.com/gpucores</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">10</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 算力的 10%，以 1% 为步进</span><br></div></code></pre></div></div>
<p>这些对默认调度器而言没有任何含义。扩展资源在 Kubernetes 眼里只是不透明的整数：它能从一个节点的总量里减掉，仅此而已。它不知道 <code>gpumem</code> 和 <code>gpucores</code> 必须落在同一张物理卡上，也不知道两张各要 8000 MiB 的 Pod 能塞进一张 24 GiB 的卡，而第三张要 12000 MiB 的 Pod 却塞不下（HAMi 的调度器会在过滤阶段以 <code>CardInsufficientMemory</code> 事件拒绝它）。于是 HAMi 把 Pod 经由一个变更型 Webhook 引到自己的调度器扩展器，由它过滤节点、为各张卡打分、挑出具体的设备 UUID，并把决策记录在 API 唯一容得下的地方，即一条注解里：</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">hami.io/vgpu-devices-allocated: GPU-&lt;device-uuid&gt;,NVIDIA,8000,10:</span><br></div></code></pre></div></div>
<p>节点上的设备插件随后在 <code>Allocate()</code> 里读取这条注解，向容器注入 <code>CUDA_DEVICE_MEMORY_LIMIT_0=8000m</code> 与 <code>CUDA_DEVICE_SM_LIMIT=10</code>，并预加载 <code>libvgpu.so</code>，让限制得以生效。这套机制是可行的，而且在生产规模上被验证过（DaoCloud 在 10+ 数据中心、10000+ 张 GPU 上跑着它）。</p>
<p>注意这套设计里有多少是「绕路」：Webhook 之所以存在，是因为调度器解析不了请求；注解之所以存在，是因为 API 没有「哪张卡、多少量」这个字段；而调度器与 kubelet 之间的整套契约，全押在只有 HAMi 组件能读懂的字符串格式上。</p>
<p>那个年代的每一个碎片化 GPU 项目都做了同样的取舍，各有各的私有注解方言。这正是 DRA 被造出来要终结的局面。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="dra-在底层改变了什么">DRA 在底层改变了什么<a href="https://project-hami.io/zh/blog/does-kubernetes-dra-replace-hami#dra-%E5%9C%A8%E5%BA%95%E5%B1%82%E6%94%B9%E5%8F%98%E4%BA%86%E4%BB%80%E4%B9%88" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>DRA 用一套<strong>声明</strong>（claims）模型取代了整数计数，其形态刻意做成了像 PersistentVolumeClaim（PVC）的样子。<code>resource.k8s.io/v1</code> API 组里的 4 个对象承载了整个流程，每个对象各有不同的所有者：</p>
<ul>
<li class=""><strong>ResourceSlice</strong>：由设备驱动发布。按节点描述真实硬件，并附带结构化属性（型号、显存、架构），让调度器看到的不再是干瘪的计数，而是设备本身。</li>
<li class=""><strong>DeviceClass</strong>：由集群管理员编写。定义设备类别，用通用表达式语言（CEL）对上述属性做过滤。</li>
<li class=""><strong>ResourceClaim 与 ResourceClaimTemplate</strong>：由负载所有者编写。一个 Claim 按类别、选择器与约束来申请设备；一个 Template 则为每个 Pod 盖出一个 Claim，让每个副本各得其所。</li>
</ul>
<p>调度器在绑定 Pod 之前，把一个具体的设备分配给某个 Claim，结果作为带类型的 API 对象存在 Claim 的 status 里。把它和上面那条注解字符串对比一下：「哪张卡、多少量」这个决策，现在有了一等公民的安身之处：kubectl 能读、RBAC 能守、其他控制器能基于它继续构建。</p>
<p>时间线对规划很重要。核心 DRA 在 Kubernetes v1.34 中 GA，自 v1.35 起锁定开启。各扩展仍在以各自节奏毕业：优先设备列表（先要一张大卡，退而求其次两张小卡）在 v1.36 转为 stable；可分区设备（partitionable devices）和可消费容量则在 v1.36 进入 beta，截至本文写作时（2026 年 7 月）尚未稳定。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="consumable-capacity这里真正关键的那块">Consumable capacity：这里真正关键的那块<a href="https://project-hami.io/zh/blog/does-kubernetes-dra-replace-hami#consumable-capacity%E8%BF%99%E9%87%8C%E7%9C%9F%E6%AD%A3%E5%85%B3%E9%94%AE%E7%9A%84%E9%82%A3%E5%9D%97" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>光有核心 DRA，还不足以给出 HAMi 式的共享。它的基线共享模型是「多个 Pod 引用同一个 ResourceClaim」，即它们共享同一份分配，而非各得一份被记账的切片。真正能与 HAMi 模型对应上的，是<strong>可消费容量</strong>（consumable capacity）：v1.34 以 alpha 引入（<code>DRAConsumableCapacity</code> 特性门控），v1.36 起 beta 并默认开启。</p>
<p>它带来两样东西：其一，驱动可以给一台设备打上 <code>allowMultipleAllocations</code>，声明来自不同命名空间的独立 Claim 可以同时落在其上；其二，一个 Claim 可以携带<strong>容量请求</strong>，申请设备上某个具名资源的一定数量，而不是整台设备全要。于是调度器对显存做的事，就如同它一直对节点内存做的那样：记账，并保证已授予的容量之和绝不超出设备所声明的总量。</p>
<p>把这些和 HAMi 的扩展资源对齐，映射几乎是机械式的：<code>nvidia.com/gpumem: 8000</code> 变成对 memory 的容量请求；<code>nvidia.com/gpucores: 10</code> 变成对 compute 的容量请求；HAMi 调度器扩展器的过滤步骤（「这张卡还有没有 8000 MiB 未承诺？」）变成了上游调度器自己的算术。HAMi 用户所熟知的 <code>CardInsufficientMemory</code> 拒绝，则变成了一次标准的不可调度 Claim。</p>
<p>正因如此，HAMi 的维护者把 DRA 视为<strong>收敛</strong>而非竞争：上游 Kubernetes 采纳的，正是那个变通方案一路以来所实现的同一套模型，而 HAMi 的 2026 路线图也把「完成 DRA 标准适配」列为目标之一。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="调度仍然只是问题的一半">调度，仍然只是问题的一半<a href="https://project-hami.io/zh/blog/does-kubernetes-dra-replace-hami#%E8%B0%83%E5%BA%A6%E4%BB%8D%E7%84%B6%E5%8F%AA%E6%98%AF%E9%97%AE%E9%A2%98%E7%9A%84%E4%B8%80%E5%8D%8A" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>接下来这条边界，决定着你到底还需不需要 HAMi。<strong>DRA（连可消费容量在内）是一个承诺跟踪器。</strong> 它保证调度器不会把设备没有的量许出去。它对一个在运行时撕毁承诺的容器毫无办法，而 GPU 场景里真正伤人的恰恰是这种失效模式：CUDA 才不管 ResourceClaim 写了什么，一个贪婪的 <code>cudaMalloc()</code> 循环会高高兴兴地把邻居正指望的那块显存吃掉。</p>
<p><strong>强制执行（enforcement）是 HAMi 的第二项职责，它住在 <a href="https://github.com/Project-HAMi/HAMi-core" target="_blank" rel="noopener noreferrer" class="">HAMi-core</a> 里。</strong> HAMi-core 是一个 C 库（<code>libvgpu.so</code>），被预加载进容器，拦截 CUDA 与 NVML 调用，从用户态施加已授予的限制。这套行为在任何共享卡上都容易验证：给两个 Pod 各 8000 MiB 的授权，再让其中一个故意越过限额分配。违规者会恰好在它的 8000 MiB 边界处抛出 CUDA OOM，而邻居毫发无损地继续跑，哪怕物理卡上还有空闲显存。配额才是那个限流器，逐容器生效，而这正是一个多团队共享集群所需要的东西。</p>
<p>我同样要诚实地指出它的天花板：这是借助库拦截实现的<strong>软件级强制</strong>。绕过预加载的工作负载（对驱动静态链接、设置 <code>CUDA_DISABLE_CONTROL</code>、容器套容器）能逃逸出去。对抗性的多租户场景需要硬件分区（NVIDIA MIG，HAMi 也能动态调度它）；而对于协作团队共享昂贵显卡的场景，拦截是粒度上的赢家：1 MiB 显存步进、1% 算力步进，对阵 MIG 的固定档位。</p>
<p>DRA 里没有任何东西替代这一层。DRA 驱动的职责止步于容器设备接口（CDI）：告诉运行时挂载哪些设备节点、设置哪些环境。进程开始调用 CUDA 之后发生的事，按设计不在其范围内。所以现实的架构是把两者配成一对：DRA 做请求与调度语言，HAMi-core 做运行时肌肉，中间再有一个驱动负责把前者翻译成后者。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="hami-如今交付的-dra-技术栈">HAMi 如今交付的 DRA 技术栈<a href="https://project-hami.io/zh/blog/does-kubernetes-dra-replace-hami#hami-%E5%A6%82%E4%BB%8A%E4%BA%A4%E4%BB%98%E7%9A%84-dra-%E6%8A%80%E6%9C%AF%E6%A0%88" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>那个驱动是存在的，相关的工作散落在值得逐一认识的 3 个仓库里，因为它们各自解决不同的运维问题。</p>
<p><a href="https://github.com/Project-HAMi/k8s-dra-driver" target="_blank" rel="noopener noreferrer" class="">k8s-dra-driver</a> 是地基：一个 DRA 驱动，把每张 GPU 的显存与算力以可消费容量形式发布到 ResourceSlice 里，运行在节点上解析分配的 kubelet 插件，并通过 CDI 把容器接好，同时挂上 HAMi-core 的强制执行。项目称其为「首个启用可消费容量的、面向 NVIDIA GPU 的开源 DRA 驱动」。</p>
<p><a href="https://github.com/Project-HAMi/HAMi-DRA" target="_blank" rel="noopener noreferrer" class="">HAMi-DRA</a> 回答的是每个平台团队紧接着会问的问题：那些已经按 <code>nvidia.com/gpu</code> 和 <code>nvidia.com/gpumem</code> 写好的成百上千份清单怎么办？它是一个变更型准入 Webhook，把入站 Pod 里经典的扩展资源剥掉，并即时生成等价的 ResourceClaim，同时保留大家熟悉的 UUID 与设备类型注解以便精准定位。它翻译的资源名是可配置的（chart values 里的 <code>resourceName</code>、<code>resourceMem</code>、<code>resourceCores</code>），所以被改名或厂商专属的资源也能继续工作。</p>
<p>如果这套模式听着眼熟，那确实应该眼熟：HAMi 的传统流水线也是从变更型 Webhook 起步的。而这背后的回报远不止于向后兼容：因为 Webhook 发出的是一份标准 ResourceClaim，并把调度交给它的所有者，<strong>HAMi-DRA 可以不着一处补丁地落入跑着 Volcano、KAI Scheduler 或任何其他调度器的集群。</strong> 传统流水线不得不把自己的扩展器注入调度路径，这意味着每个第三方调度器都需要 HAMi 专属的集成。DRA 抹掉了这种耦合：请求是一个任何 DRA 感知调度器都认得的对象，于是 HAMi 再也不必改动上游。拦截点在迁移中存活了下来；变的只是写进 Pod 里的内容。</p>
<p>伴随 HAMi v2.9，HAMi-DRA v0.2.0 被宣布为生产就绪，发布线随后推进到 v0.2.1。它的平台清单也越过了 NVIDIA，扩展到昇腾（Ascend）与燧原（Enflame），海光 DCU 则通过海光自家的 k8s-dcu-dra-driver 提供文档。</p>
<p>HAMi 本身自 v2.8 起把 <a class="" href="https://project-hami.io/zh/docs/installation/how-to-use-hami-dra">DRA 模式列为一种安装选项</a>，可观测性故事也一并延续：DRA monitor 组件默认开启，通过 Prometheus（端口 31995）暴露逐容器的设备指标，于是那些基于 HAMi exporter 搭建的仪表盘能熬过这次切换。孵化公告也把「监控 DRA 消费」作为一等关切交给了团队。</p>
<p>把调度交给集群自己的调度器，有一个值得点名的代价：<strong>HAMi-DRA 不自带调度器，因此无法做拓扑感知的放置决策。</strong> 假设某节点上两张 GPU 由 NVLink 相连（GPU0 与 GPU1、GPU2 高带宽可达，但与 GPU3 不可达），那么一个要两张 GPU 的 Claim，理应优先落在 NVLink 相连的一对上，而不是被切到慢路径两侧。Webhook 表达不了这个，它交给调度器的是计数和容量，而非拓扑约束。对于卡间带宽敏感的负载，你要么留在一个能建模拓扑的调度器上，要么接受放置对带宽「无感」。</p>
<p>安装 Webhook 与驱动是一次 Helm 发布，需先备好 cert-manager 以签发 Webhook 的服务证书（或自带 <code>certs.custom.crt</code> 与 <code>certs.custom.key</code>）：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm repo </span><span class="token function" style="color:rgb(130, 170, 255)">add</span><span class="token plain"> hami-dra https://project-hami.github.io/HAMi-DRA</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm repo update</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm </span><span class="token function" style="color:rgb(130, 170, 255)">install</span><span class="token plain"> hami-dra hami-dra/hami-dra</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 若 NVIDIA 驱动装在宿主机上而非通过 GPU Operator，则追加：</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># --set drivers.nvidia.containerDriver=false</span><br></div></code></pre></div></div>
<p>若由 GPU Operator 管理驱动，请先用 <code>devicePlugin.enabled=false</code> 安装它，因为 DRA 栈取代了设备插件的职责。发布稳定后，你会看到 3 个 Pod：kubelet 插件、Webhook 与 monitor。<code>kubectl get resourceslice</code> 确认驱动在发布 GPU 容量；一旦负载落上，<code>kubectl get resourceclaim</code> 就能看到调度器所做的分配，它现在在 API 里，而不是一条注解里。</p>
<p>前置条件比传统模式更严，而且每一项都是真门槛：</p>
<ul>
<li class=""><strong>Kubernetes v1.34 或更高，并启用 <code>DRAConsumableCapacity</code>。</strong> 在 v1.34 与 v1.35 上该门控是 alpha 且默认关闭，需在控制面设置，这就把不开放 API server 标志的托管集群挡在了门外。该门控在 v1.36 升入 beta 并默认开启，悄悄移除了最大的采纳障碍。</li>
<li class=""><strong>支持 CDI 的运行时。</strong> 启用 CDI 的 containerd 或 CRI-O，外加 NVIDIA 驱动 440 或更高。</li>
<li class=""><strong>面向你所选芯片的 DRA 驱动。</strong> NVIDIA 是成熟路径；昇腾、燧原与海光 DCU 正在路上。HAMi 的传统模式覆盖的清单要长得多（截至 v2.9 有 12+ 个设备家族，含寒武纪 MLU、天数、摩尔线程、昆仑芯、AWS Neuron、Vastai 等，通过各厂商的设备插件接入）。在驱动清单补齐之前，异构集群仍需留在设备插件路径上。</li>
</ul>
<div class="theme-admonition theme-admonition-warning admonition_xJq3 alert alert--warning"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 16 16"><path fill-rule="evenodd" d="M8.893 1.5c-.183-.31-.52-.5-.887-.5s-.703.19-.886.5L.138 13.499a.98.98 0 0 0 0 1.001c.193.31.53.501.886.501h13.964c.367 0 .704-.19.877-.5a1.03 1.03 0 0 0 .01-1.002L8.893 1.5zm.133 11.497H6.987v-2.003h2.039v2.003zm0-3.004H6.987V5.987h2.039v4.006z"></path></svg></span>重要提醒</div><div class="admonitionContent_BuS1"><p>DRA 模式与传统设备插件模式不得在同一集群中同时运行。文档对此表述直白，原因也由上文顺理推出：两个记账人（调度器扩展器与 DRA 调度器）会各自以为拥有同一份显存，而谁也看不到对方的承诺。每个集群只选一种模式。</p></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="同一份-gpu-切片新旧两种写法">同一份 GPU 切片，新旧两种写法<a href="https://project-hami.io/zh/blog/does-kubernetes-dra-replace-hami#%E5%90%8C%E4%B8%80%E4%BB%BD-gpu-%E5%88%87%E7%89%87%E6%96%B0%E6%97%A7%E4%B8%A4%E7%A7%8D%E5%86%99%E6%B3%95" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>这种收敛，在 YAML 里最一目了然，以下两例均取自当前 <a class="" href="https://project-hami.io/zh/docs/installation/how-to-use-hami-dra">HAMi DRA 文档</a>。</p>
<p>先看 HAMi 用户写了多年的样子。要一张卡、其中 10 GiB 显存、一半算力：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">limits</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">nvidia.com/gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">nvidia.com/gpumem</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">10240</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">nvidia.com/gpucores</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">50</span><br></div></code></pre></div></div>
<p>这段在 DRA 模式下依然原样可用，Webhook 会在准入时把它改写成下面的原生形态，所以无需改动任何清单。</p>
<p>下面是同一份请求的原生写法，写成针对 HAMi DeviceClass 的 ResourceClaim：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">apiVersion</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> resource.k8s.io/v1</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">kind</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> ResourceClaim</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">metadata</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">half</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">claim</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">spec</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">devices</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">requests</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> gpu</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token key atrule">exactly</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">deviceClassName</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> hami</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">core</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">gpu.project</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">hami.io</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">allocationMode</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> ExactCount</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">count</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">capacity</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">            </span><span class="token key atrule">requests</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">              </span><span class="token key atrule">cores</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">50</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">              </span><span class="token key atrule">memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"10Gi"</span><br></div></code></pre></div></div>
<p>Pod 引用该 Claim，而不是列出资源 limits：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">apiVersion</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> v1</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">kind</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Pod</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">metadata</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">test</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">dra</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">native</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">spec</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">containers</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> cuda</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">image</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> nvidia/cuda</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">13.0.1</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">base</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">ubi9</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">command</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">[</span><span class="token string" style="color:rgb(195, 232, 141)">"sleep"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"3600"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">]</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token key atrule">claims</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> gpu</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">resourceClaims</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> gpu</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">resourceClaimName</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">half</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">claim</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">restartPolicy</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Never</span><br></div></code></pre></div></div>
<p>映射是直接的：<code>nvidia.com/gpu</code> 变成设备计数，<code>gpumem</code> 变成 memory 容量请求，<code>gpucores</code> 变成 <code>cores</code>。原生形态更长，但这种冗长正是要点所在。旧的三行块对调度器不透明，所以 HAMi 才需要 Webhook、扩展器和注解去作用于它。ResourceClaim 用一种上游调度器读得懂、RBAC 守得住、kubectl 看得见的形态，把同一件事讲出来，无需任何私有注解方言。</p>
<p>无论走哪条路，运行容器里的 <code>nvidia-smi</code> 都会报出 10240 MiB（所授予的那个切片，而非物理卡），因为 HAMi-core 做的拦截与过去并无二致。请求语言变了；运行时契约没变。</p>
<p>在做决策树之前，先用一张表把整个对比尽收眼底：</p>
<table><thead><tr><th></th><th>设备插件模式</th><th>DRA 模式</th></tr></thead><tbody><tr><td>请求语言</td><td>扩展资源，不透明整数</td><td>ResourceClaim 容量请求，带类型 API</td></tr><tr><td>调度决策</td><td>HAMi 调度器扩展器</td><td>默认 kube-scheduler</td></tr><tr><td>分配记录</td><td>仅 HAMi 读取的注解字符串</td><td>ResourceClaim status，kubectl 与 RBAC 可见</td></tr><tr><td>准入 Webhook</td><td>必需，把 Pod 引到扩展器</td><td>可选，HAMi-DRA 翻译遗留 YAML</td></tr><tr><td>Kubernetes 版本</td><td>任何受支持版本</td><td>v1.34+ 需特性门控；v1.36 起默认开启</td></tr><tr><td>运行时前置条件</td><td>标准容器运行时</td><td>支持 CDI 的 containerd 或 CRI-O，NVIDIA 驱动 440+</td></tr><tr><td>厂商覆盖</td><td>12+ 个设备家族</td><td>NVIDIA 成熟；昇腾、燧原、海光 DCU 路上</td></tr><tr><td>强制执行</td><td>HAMi-core（<code>libvgpu.so</code>）</td><td>同一 HAMi-core，不变</td></tr><tr><td>可观测性</td><td>HAMi exporters</td><td>DRA monitor，Prometheus 端口 31995</td></tr><tr><td>生产历练</td><td>16 个版本，DaoCloud 10000+ GPU</td><td>HAMi-DRA v0.2.1，自 v2.9 起生产就绪</td></tr><tr><td>共存</td><td>绝不在同集群内并存</td><td>同一约束</td></tr></tbody></table>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="2026-年年中如何选择">2026 年年中，如何选择<a href="https://project-hami.io/zh/blog/does-kubernetes-dra-replace-hami#2026-%E5%B9%B4%E5%B9%B4%E4%B8%AD%E5%A6%82%E4%BD%95%E9%80%89%E6%8B%A9" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>我的判断，来自一个既跑着传统流水线、又一路追踪 DRA 进展的人：</p>
<ul>
<li class=""><strong>托管 Kubernetes，且在 v1.34/v1.35 上拿不到特性门控权限，或任何更老的版本：</strong> 传统模式，毫不犹豫。这是一条带着多年生产历练的路：16 个版本，背后还有独立案例研究报告其带来 3 倍利用率提升。</li>
<li class=""><strong>多厂商加速器集群：</strong> 传统模式，直到 DRA 驱动覆盖到你的芯片。这是 HAMi 最宽的护城河，也是 DRA 最后才触及的部分。</li>
<li class=""><strong>v1.34+ 上你能掌控控制面的 NVIDIA 集群，以及任何 v1.36 用户：</strong> 现在就在 staging 环境里把 DRA 模式立起来。先用 HAMi-DRA Webhook，这样无需改任何清单；对照你真实的负载组合观察 <code>kubectl get resourceclaim</code>，让结果来决定你的生产时间线。你最终会落在一个上游调度器原生就懂的 API 上，而随着优先列表等扩展趋于稳定，其收益会不断复利。</li>
</ul>
<p>仍剩三个诚实的告诫：可消费容量尚未在 upstream 达到 stable；k8s-dra-driver 自己的 Helm chart 仍标注为 work in progress；DRA 侧的厂商覆盖只是传统模式的一小部分。这些都没有被藏着掖着，且假以时日皆可补齐，方向在两边都已定下，HAMi 的 2026 路线图亦把「完成 DRA 标准适配」列为目标。</p>
<p>Kubernetes 上的 GPU 共享，最初是建在一个只会数数的 API 的缝隙里的。DRA 补上了那条缝，HAMi 留住了那身肌肉。有史以来第一次，请求、调度与强制执行，说起了同一种语言。</p>
<p>感谢以下同事在终稿审阅中给予的宝贵反馈：Mengxuan Li（HAMi）、Jimmy Song（HAMi）、Sarah Christoff（Linkerd）。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="参考资料">参考资料<a href="https://project-hami.io/zh/blog/does-kubernetes-dra-replace-hami#%E5%8F%82%E8%80%83%E8%B5%84%E6%96%99" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class=""><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/" target="_blank" rel="noopener noreferrer" class="">Kubernetes 文档，Dynamic Resource Allocation</a></li>
<li class=""><a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/" target="_blank" rel="noopener noreferrer" class="">Kubernetes 文档，Feature Gates reference</a></li>
<li class=""><a href="https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/" target="_blank" rel="noopener noreferrer" class="">Kubernetes 文档，Dynamic Admission Control</a></li>
<li class=""><a href="https://kubernetes.io/blog/2025/09/01/kubernetes-v1-34-dra-updates/" target="_blank" rel="noopener noreferrer" class="">Kubernetes 博客，Kubernetes v1.34: DRA has graduated to GA</a></li>
<li class=""><a href="https://kubernetes.io/blog/2025/09/18/kubernetes-v1-34-dra-consumable-capacity/" target="_blank" rel="noopener noreferrer" class="">Kubernetes 博客，Kubernetes v1.34: DRA Consumable Capacity</a></li>
<li class=""><a href="https://www.cncf.io/blog/2026/07/15/hami-becomes-a-cncf-incubating-project/" target="_blank" rel="noopener noreferrer" class="">CNCF 博客，HAMi becomes a CNCF incubating project</a></li>
<li class=""><a href="https://github.com/Project-HAMi/HAMi" target="_blank" rel="noopener noreferrer" class="">Project HAMi，HAMi 仓库</a></li>
<li class=""><a href="https://github.com/Project-HAMi/HAMi-core" target="_blank" rel="noopener noreferrer" class="">Project HAMi，HAMi-core 仓库</a></li>
<li class=""><a href="https://github.com/Project-HAMi/HAMi-DRA" target="_blank" rel="noopener noreferrer" class="">Project HAMi，HAMi-DRA 仓库</a></li>
<li class=""><a href="https://github.com/Project-HAMi/k8s-dra-driver" target="_blank" rel="noopener noreferrer" class="">Project HAMi，k8s-dra-driver 仓库</a></li>
<li class=""><a class="" href="https://project-hami.io/zh/docs/installation/how-to-use-hami-dra">HAMi 文档，How to use HAMi DRA</a></li>
</ul>]]></content>
        <author>
            <name>Mesut Oezdil</name>
            <uri>https://www.linkedin.com/in/mesut-oezdil/</uri>
        </author>
        <category label="HAMi" term="HAMi"/>
        <category label="DRA" term="DRA"/>
        <category label="GPU 共享" term="GPU 共享"/>
        <category label="Kubernetes" term="Kubernetes"/>
        <category label="调度" term="调度"/>
        <category label="云原生" term="云原生"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[用 Volcano + HAMi-core 软切分昇腾 vNPU：原理与真机验证]]></title>
        <id>https://project-hami.io/zh/blog/volcano-ascend-vnpu-soft-slicing</id>
        <link href="https://project-hami.io/zh/blog/volcano-ascend-vnpu-soft-slicing"/>
        <updated>2026-08-17T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Volcano 的 HAMi 模式 deviceshare 如何把批量调度与 HAMi-core 运行时隔离结合到昇腾 NPU 上，在昇腾 310P3 ARM 服务器上从源码编译镜像开始，端到端验证容器内显存切片、binpack 共卡与容器级监控指标。]]></summary>
        <content type="html"><![CDATA[<p><a href="https://github.com/volcano-sh/volcano" target="_blank" rel="noopener noreferrer" class="">Volcano</a> 是很多 AI 集群的批量调度器首选，HAMi-core 则是让共享加速器"守规矩"的运行时。本文关注两者在昇腾硬件上的交汇点：在 <strong>Volcano 调度器下运行 <code>hami-vnpu-core</code> 软切分的 vNPU</strong>，让批量调度语义（队列、Gang、binpack）与容器级隔离（在昇腾 API 层强制生效的显存与算力上限）协同工作。</p>
<p>我们在一台昇腾 310P3 aarch64 服务器的单节点 Kubernetes 集群上验证了完整链路：源码编译 Volcano 镜像、部署官方 <a href="https://github.com/Project-HAMi/ascend-device-plugin" target="_blank" rel="noopener noreferrer" class="">ascend-device-plugin</a> v1.4.0 镜像，并确认申请 8192 MiB 切片的容器恰好只能看到这么多显存；同时第二个 Pod 以 binpack 方式落到同一张物理卡上、拿到独立切片，插件的 Prometheus 端点也如实上报了两个容器的配额。完整步骤（含每条命令与真实输出）见 <a class="" href="https://project-hami.io/zh/tutorials/labs/volcano-ascend-vnpu">实验 13：用 Volcano + HAMi-core 软切分昇腾 310P3 vNPU</a>。</p>
<p>这个话题里混着好几个经常被混为一谈的概念，所以本文先把层次分开：vNPU 是什么、硬切分和软切分有何不同、Volcano 集成相比已有的 HAMi 调度器路径到底新增了什么。</p>
<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"></path></svg></span>关于本文中的输出</div><div class="admonitionContent_BuS1"><p>本文所有输出均采集自真实的昇腾 310P3 物理服务器，截至撰写本文时已在真机上验证：麒麟 V10 aarch64 节点、2× 昇腾 310P3（驱动/npu-smi 25.5.1）、Kubernetes v1.28.15、containerd 1.7.1。其他集群中的 UUID、IP、Pod 后缀会不同；请对比组件名、调度位置与测量值。</p></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="先把-npuvnpu硬切分软切分分清楚">先把 NPU、vNPU、硬切分、软切分分清楚<a href="https://project-hami.io/zh/blog/volcano-ascend-vnpu-soft-slicing#%E5%85%88%E6%8A%8A-npuvnpu%E7%A1%AC%E5%88%87%E5%88%86%E8%BD%AF%E5%88%87%E5%88%86%E5%88%86%E6%B8%85%E6%A5%9A" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>vNPU、硬切分、软切分、HAMi 模式、Volcano 支持分别处在不同的层次，混在一起看就会显得像同一件事。它们各自回答不同的问题：</p>
<table><thead><tr><th style="text-align:left">层次</th><th style="text-align:left">回答的问题</th></tr></thead><tbody><tr><td style="text-align:left">NPU / vNPU</td><td style="text-align:left">你拿到的是什么设备：vNPU 是从物理 NPU 划出、提供给容器的逻辑设备</td></tr><tr><td style="text-align:left">硬切分 / 软切分</td><td style="text-align:left">这个虚拟设备如何实现隔离</td></tr><tr><td style="text-align:left">HAMi-core</td><td style="text-align:left">谁在容器内执行软切分配额</td></tr><tr><td style="text-align:left">HAMi / Volcano</td><td style="text-align:left">由谁决定 Pod 用哪张卡、用多少资源</td></tr></tbody></table>
<p>NPU 是物理设备，例如服务器里真实存在的一张昇腾 310P3；vNPU 是从物理 NPU 上划出来、提供给容器使用的逻辑设备（一张 21.5 GiB 的卡可以划成 8 GiB 加 8 GiB 两份切片，剩余约 5.5 GiB）。但“vNPU”只表示结果，并不说明实现：切片既可能由昇腾驱动的虚拟化能力切出来，也可能由 HAMi-core 用软件模拟出来。这也是本话题最容易产生歧义的地方：本文所说的“软切分 vNPU”是 Kubernetes/HAMi 视角下的逻辑切片，并不是通过 <code>npu-smi ... create-vnpu</code> 创建的昇腾硬件 vNPU。</p>
<p><strong>硬切分</strong>由昇腾驱动/固件的虚拟化能力完成，用户只能从预定义模板中选择：<code>vir05_1c_16g</code> 代表固定数量的 AI Core、AI CPU 和 16 GiB 显存，创建后设备层会产生一个真正的 vNPU 实例。可用 <code>npu-smi info -t template-info</code> 查询芯片支持的模板，完整实践见昇腾社区的<a href="https://www.hiascend.com/developer/techArticles/20251212-1" target="_blank" rel="noopener noreferrer" class="">《NPU 卡虚拟化硬切分参考实践》</a>。</p>
<p><strong>软切分</strong>不在硬件里真正创建 vNPU，而是让多个容器共享同一张物理 NPU，在容器内注入 <code>libvnpu.so</code> 拦截应用对昇腾运行时 API 的调用并记账：</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">应用</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  ↓ 昇腾 API 调用</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">libvnpu.so 拦截并记账</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  ↓ 只允许使用分配到的显存和算力</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">昇腾驱动</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  ↓</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">物理 NPU</span><br></div></code></pre></div></div>
<p>以 8192 MiB 配额为例：应用查询设备时只看到 8192 MiB，申请显存由 <code>libvnpu.so</code> 记账，超过配额的请求被运行时拦截层阻止，多个容器可以共享同一张物理卡。官方 <a href="https://github.com/Project-HAMi/ascend-device-plugin" target="_blank" rel="noopener noreferrer" class="">ascend-device-plugin 文档</a>也明确区分了“模板硬切分”与“运行时拦截的软切分”两种模式。两者对比：</p>
<table><thead><tr><th style="text-align:left"></th><th style="text-align:left">硬切分</th><th style="text-align:left">软切分</th></tr></thead><tbody><tr><td style="text-align:left">隔离边界</td><td style="text-align:left">设备虚拟化层强制，更强</td><td style="text-align:left">软件运行时拦截，不等于 SR-IOV 级硬件边界</td></tr><tr><td style="text-align:left">规格</td><td style="text-align:left">仅厂商模板，例如 8 GiB、16 GiB 档位</td><td style="text-align:left">任意 MiB、算力比例</td></tr><tr><td style="text-align:left">切分单元</td><td style="text-align:left">AI Core、AI CPU、显存、DVPP</td><td style="text-align:left">显存与算力配额</td></tr><tr><td style="text-align:left">前置条件</td><td style="text-align:left">芯片/驱动需支持对应模板</td><td style="text-align:left">依赖 <code>libvnpu.so</code> 注入与驱动兼容，当前仅支持 ARM</td></tr></tbody></table>
<p>可以把两者理解成：<strong>硬切分是在房子里真正砌墙；软切分是大家共用房子，但每个门口都有一个严格记账和限流的管理员。</strong></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="volcano-调度昇腾-vnpu-的两种方式">Volcano 调度昇腾 vNPU 的两种方式<a href="https://project-hami.io/zh/blog/volcano-ascend-vnpu-soft-slicing#volcano-%E8%B0%83%E5%BA%A6%E6%98%87%E8%85%BE-vnpu-%E7%9A%84%E4%B8%A4%E7%A7%8D%E6%96%B9%E5%BC%8F" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>Volcano 调度昇腾虚拟 NPU 有<strong>两种不同方式</strong>，很容易混淆。先把它讲清楚，能省掉好几个小时的排错：</p>
<table><thead><tr><th style="text-align:left"></th><th style="text-align:left">MindCluster 模式</th><th style="text-align:left">HAMi 模式</th></tr></thead><tbody><tr><td style="text-align:left">Volcano 开关</td><td style="text-align:left"><code>deviceshare.AscendMindClusterVNPUEnable</code></td><td style="text-align:left"><code>deviceshare.AscendHAMiVNPUEnable</code></td></tr><tr><td style="text-align:left">提供方</td><td style="text-align:left">Volcano 原生昇腾插件</td><td style="text-align:left"><a href="https://github.com/Project-HAMi/ascend-device-plugin" target="_blank" rel="noopener noreferrer" class="">Project-HAMi/ascend-device-plugin</a></td></tr><tr><td style="text-align:left">模板</td><td style="text-align:left"><code>vir04_3c_ndvpp</code>（带 <code>dvpp</code> 维度）</td><td style="text-align:left"><code>vir05_1c_16g</code>（仅 <code>memory</code>/<code>aiCore</code>/<code>aiCPU</code> 字段）</td></tr><tr><td style="text-align:left">切分方式</td><td style="text-align:left">驱动模板（硬切分）</td><td style="text-align:left">默认走驱动模板（硬切分），Pod 设置 <code>huawei.com/vnpu-mode: hami-core</code> 后走 <code>hami-core</code> 软切分</td></tr><tr><td style="text-align:left">资源名</td><td style="text-align:left"><code>huawei.com/npu-core</code></td><td style="text-align:left"><code>huawei.com/Ascend310P</code>、<code>-memory</code></td></tr></tbody></table>
<p>本文讲的是 <strong>HAMi 模式</strong>下的 <strong><code>hami-vnpu-core</code> 软切分</strong>。注意，HAMi 模式并不等于软切分模式：同一个 ascend-device-plugin 同时支持模板硬切分与 hami-vnpu-core 软切分，由 Pod 的注解选择路径。实验中的 Pod 设置了该注解，因此走的是软切分路径，也是两种 Volcano 模式里唯一做运行时拦截的一种：不是把卡预先切成固定的虚拟化模板，而是在用户态拦截昇腾调用，在运行时按容器强制显存与算力上限。Volcano 决定哪个 Pod 拿到哪个切片，HAMi-core 让这个决定真正生效。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="volcano-集成到底新增了什么">Volcano 集成到底新增了什么<a href="https://project-hami.io/zh/blog/volcano-ascend-vnpu-soft-slicing#volcano-%E9%9B%86%E6%88%90%E5%88%B0%E5%BA%95%E6%96%B0%E5%A2%9E%E4%BA%86%E4%BB%80%E4%B9%88" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>先纠正一个名字：<strong>HAMi-core</strong> 是这一类容器内运行时隔离技术的统称，最早主要指 NVIDIA 的 <code>libvgpu.so</code>；<strong>hami-vnpu-core</strong> 是专门面向昇腾 NPU 的实现，实际注入的是 <code>libvnpu.so</code>。</p>
<p>昇腾软切分能力本身并不是新东西。相关支持在 2026 年 4 月已经进入代码（<a href="https://github.com/Project-HAMi/ascend-device-plugin/pull/61" target="_blank" rel="noopener noreferrer" class="">ascend-device-plugin 集成 hami-vnpu-core</a>、<a href="https://github.com/Project-HAMi/HAMi/pull/1771" target="_blank" rel="noopener noreferrer" class="">HAMi 增加昇腾 ResourceCoreName 和软切分调度支持</a>），在 HAMi 2.9 中链路已经可用：HAMi 自己的调度器分配昇腾切片，<code>ascend-device-plugin</code> 完成设备挂载，<code>hami-vnpu-core</code> 在容器内执行显存和算力限制。</p>
<p>即将发布的 HAMi 2.10 中的 Volcano 集成没有重新发明软切分，而是把"分配者"从 HAMi Scheduler 换成了 Volcano，底下两层原样复用：</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">HAMi 2.9:</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">HAMi Scheduler → ascend-device-plugin → hami-vnpu-core → NPU</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">HAMi 2.10 / Volcano 集成：</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">Volcano Scheduler → ascend-device-plugin → hami-vnpu-core → NPU</span><br></div></code></pre></div></div>
<table><thead><tr><th style="text-align:left">层次</th><th style="text-align:left">HAMi 2.9 路径</th><th style="text-align:left">Volcano 集成路径</th></tr></thead><tbody><tr><td style="text-align:left">调度器</td><td style="text-align:left">HAMi Scheduler</td><td style="text-align:left">Volcano Scheduler</td></tr><tr><td style="text-align:left">设备发现/挂载</td><td style="text-align:left">ascend-device-plugin</td><td style="text-align:left">同一个 ascend-device-plugin</td></tr><tr><td style="text-align:left">软切分执行</td><td style="text-align:left">hami-vnpu-core（<code>libvnpu.so</code>）</td><td style="text-align:left">同一个 hami-vnpu-core</td></tr><tr><td style="text-align:left">显存、算力隔离</td><td style="text-align:left">已支持</td><td style="text-align:left">复用原有能力</td></tr><tr><td style="text-align:left">队列、Gang 调度</td><td style="text-align:left">不是重点</td><td style="text-align:left">Volcano 提供</td></tr><tr><td style="text-align:left">binpack / spread</td><td style="text-align:left">HAMi 策略</td><td style="text-align:left">Volcano deviceshare 策略</td></tr><tr><td style="text-align:left">监控及软硬混合管理</td><td style="text-align:left">相对早期</td><td style="text-align:left">2.10 进一步补全</td></tr></tbody></table>
<p>Volcano 现在能理解这些 HAMi 昇腾资源，并决定：哪个 Pod 使用哪张物理 NPU、多个 Pod 是否 binpack 到同一张卡、整组训练 Pod 是否满足 Gang 条件、使用哪个队列、优先级和抢占策略。因此准确的说法是：<strong>这次集成完成的是"Volcano 调度昇腾 HAMi-core 软切分资源"，并补充监控及软硬切分混合管理；昇腾软切分能力本身早已存在。</strong></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="集成是如何工作的">集成是如何工作的<a href="https://project-hami.io/zh/blog/volcano-ascend-vnpu-soft-slicing#%E9%9B%86%E6%88%90%E6%98%AF%E5%A6%82%E4%BD%95%E5%B7%A5%E4%BD%9C%E7%9A%84" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>这条链路有三个分工：</p>
<ul>
<li class=""><strong>Volcano 的 <code>deviceshare</code> 插件</strong>从 <code>hami-scheduler-device</code> ConfigMap（配合 <code>AscendHAMiVNPUEnable: "true"</code>）读取 vNPU 规格，并按 <code>binpack</code> 或 <code>spread</code> 策略决定每个 Pod 由哪个节点、哪张卡服务。</li>
<li class=""><strong><code>ascend-device-plugin</code> DaemonSet</strong> 向节点注册 <code>huawei.com/Ascend310P</code>（卡数）与 <code>huawei.com/Ascend310P-memory</code>（MiB）扩展资源，并把 HAMi-core 资产（<code>libvnpu.so</code> 和 <code>ld.so.preload</code>）拷贝到宿主机 <code>/usr/local/hami-vnpu-core/</code>。</li>
<li class=""><strong>HAMi-core（<code>libvnpu.so</code>）</strong> 经 Ascend Docker Runtime 的 preload 机制注入业务容器，强制执行调度器选定的切片。</li>
</ul>
<figure class="mermaid-figure"><figcaption>Volcano HAMi 模式昇腾 vNPU 软切分链路</figcaption></figure>
<p>Pod 侧的契约很简洁：<code>schedulerName: volcano</code>、<code>runtimeClassName: ascend</code>、注解 <code>huawei.com/vnpu-mode: hami-core</code>，以及对两个扩展资源的 limits。少了这个注解，Pod 会退回模板路径，在纯软切分节点上可能一直 Pending。</p>
<p><a href="https://github.com/Project-HAMi/hami-vnpu-core" target="_blank" rel="noopener noreferrer" class="">hami-vnpu-core</a> 的 <code>libvnpu.so</code> 在容器内通过 <code>/etc/ld.so.preload</code> 预加载，经 <code>/hami-shared-region</code> 共享内存区通信，并通过 <code>NPU_MEM_QUOTA</code> 等环境变量拿到强制配额。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="基本步骤">基本步骤<a href="https://project-hami.io/zh/blog/volcano-ascend-vnpu-soft-slicing#%E5%9F%BA%E6%9C%AC%E6%AD%A5%E9%AA%A4" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>以下步骤都在昇腾 310P3 服务器的单节点 Kubernetes 1.28 集群上执行，完整命令与真实输出见实验 13，核心流程如下：</p>
<ol>
<li class="">
<p><strong>准备节点。</strong> 驱动/npu-smi ≥ 25.5，安装 ascend-docker-runtime，并为节点打上 <code>ascend=on</code> 标签。</p>
</li>
<li class="">
<p><strong>安装 Volcano ≥ 1.16。</strong> 软切分要求 1.16，而撰写本文时还没有稳定的 1.16（最新稳定版为 v1.15.1，仅有 <code>1.16.0-alpha.1</code> chart），因此验证时按实验 13 从源码编译了 Volcano master（commit <code>7d9504320</code>）。等稳定版本发布后，直接用 Helm 安装即可跳过编译：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm repo </span><span class="token function" style="color:rgb(130, 170, 255)">add</span><span class="token plain"> volcano-sh https://volcano-sh.github.io/helm-charts</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm </span><span class="token function" style="color:rgb(130, 170, 255)">install</span><span class="token plain"> volcano volcano-sh/volcano </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--namespace</span><span class="token plain"> volcano-system --create-namespace </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--version</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1.16</span><span class="token plain">.0</span><br></div></code></pre></div></div>
</li>
<li class="">
<p><strong>开启 HAMi 模式 deviceshare。</strong> 覆盖 <code>volcano-scheduler-configmap</code>，让 <code>deviceshare</code> 插件带上 <code>AscendHAMiVNPUEnable: "true"</code>、<code>SchedulePolicy: binpack</code>、<code>KnownGeometriesCMName: hami-scheduler-device</code>，然后重启调度器。</p>
</li>
<li class="">
<p><strong>以软切分模式部署插件。</strong> 应用 <code>ascend</code> RuntimeClass，在 <code>hami-scheduler-device</code> ConfigMap 中打开 <code>hamiVnpuCore: true</code>，并在 <code>hami-device-node-config</code> 中为节点打开 <code>hami-vnpu-core: true</code>，再部署 DaemonSet。节点随后上报 <code>huawei.com/Ascend310P</code> 与 <code>huawei.com/Ascend310P-memory</code> 资源。</p>
</li>
<li class="">
<p><strong>运行软切分 Pod。</strong> <code>schedulerName: volcano</code>、<code>runtimeClassName: ascend</code>、注解 <code>huawei.com/vnpu-mode: hami-core</code>，limits 申请 1 张卡加 8192 MiB 显存。</p>
</li>
<li class="">
<p><strong>验证。</strong> Pod 内 <code>npu-smi info</code> 显示的是切片而不是整卡，第二个 Pod 会 binpack 到同一张物理卡，插件的 <code>:9395</code> 端点导出容器级指标。</p>
</li>
</ol>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="我们验证了什么">我们验证了什么<a href="https://project-hami.io/zh/blog/volcano-ascend-vnpu-soft-slicing#%E6%88%91%E4%BB%AC%E9%AA%8C%E8%AF%81%E4%BA%86%E4%BB%80%E4%B9%88" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>宿主机上两张 310P3 均为健康状态：</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">$ npu-smi info</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">+--------------------------------------------------------------------------------------------------------+</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| npu-smi 25.5.1                                   Version: 25.5.1                                       |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">+-------------------------------+-----------------+------------------------------------------------------+</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| NPU     Name                  | Health          | Power(W)     Temp(C)           Hugepages-Usage(page) |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| Chip    Device                | Bus-Id          | AICore(%)    Memory-Usage(MB)                        |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">+===============================+=================+======================================================+</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| 4       310P3                 | OK              | NA           37                0     / 0             |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| 0       0                     | 0000:81:00.0    | 0            1848 / 21525                            |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">+===============================+=================+======================================================+</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| 5       310P3                 | OK              | NA           40                0     / 0             |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| 0       1                     | 0000:85:00.0    | 0            1849 / 21525                            |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">+===============================+=================+======================================================+</span><br></div></code></pre></div></div>
<p>插件注册后，节点上报 14 个 vNPU（2 卡 × 7，与 <code>vDeviceCount: 7</code> 一致）与 43054 MiB 可分配显存。显存数字来自芯片配置（每卡 <code>memoryAllocatable: 21527</code> MB），比 <code>npu-smi</code> 显示的 21525 MB 每卡多 2 MiB。每个测试 Pod 申请 1 个 vNPU、8192 MiB 切片：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">limits</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">huawei.com/Ascend310P</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"1"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">huawei.com/Ascend310P-memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"8192"</span><br></div></code></pre></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="1-容器只看到自己的切片">1. 容器只看到自己的切片<a href="https://project-hami.io/zh/blog/volcano-ascend-vnpu-soft-slicing#1-%E5%AE%B9%E5%99%A8%E5%8F%AA%E7%9C%8B%E5%88%B0%E8%87%AA%E5%B7%B1%E7%9A%84%E5%88%87%E7%89%87" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>第一个 Pod 内的 <code>npu-smi info</code> 显示的是 <strong>0 / 8192 MB</strong> 的设备，而不是宿主机在同一张卡上看到的 1848 / 21525 MB：</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">$ kubectl exec ascend-vnpu-check -- npu-smi info</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">[INFO limiter::supervisor] [Supervisor PID:10] won manager election</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">[INFO limiter::manager] [Manager] Registered as Global Manager #0 (PID: 10). Compute limit: 1, Memory limit: 8192, FixedShare: false</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">open global registry path is "/hami-shared-region/0_global_registry"</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">...</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| 32768   310P3                 | OK              | NA           38                0     / 0             |</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| 0       0                     | 0000:81:00.0    | 0            0    / 8192                             |</span><br></div></code></pre></div></div>
<p><code>libvnpu.so</code> 已把设备查询改写为容器的配额，注入的环境变量也印证了接线：<code>NPU_MEM_QUOTA=8192</code>、<code>NPU_GLOBAL_SHM_PATH=/hami-shared-region/0_global_registry</code>、<code>ASCEND_VISIBLE_DEVICES=0</code>。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="2-binpack-让多个-pod-共享一张物理卡">2. binpack 让多个 Pod 共享一张物理卡<a href="https://project-hami.io/zh/blog/volcano-ascend-vnpu-soft-slicing#2-binpack-%E8%AE%A9%E5%A4%9A%E4%B8%AA-pod-%E5%85%B1%E4%BA%AB%E4%B8%80%E5%BC%A0%E7%89%A9%E7%90%86%E5%8D%A1" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>第二个同规格的 Pod 落在了<strong>同一个 Bus-Id <code>0000:81:00.0</code></strong> 上，各自拥有独立的 8192 MiB 窗口，并作为 <code>Global Manager #1</code> 注册进与 Pod 1 的 <code>#0</code> 相同的共享注册表：</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">$ kubectl exec ascend-vnpu-check-2 -- npu-smi info | grep -E "Memory limit|0000"</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">[INFO limiter::manager] [Manager] Registered as Global Manager #1 (PID: 10). Compute limit: 1, Memory limit: 8192, FixedShare: false</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">| 0       0                     | 0000:81:00.0    | 0            0    / 8192                             |</span><br></div></code></pre></div></div>
<p>节点的已分配资源说的是同一件事：<code>huawei.com/Ascend310P 2</code>（共 14）、<code>huawei.com/Ascend310P-memory 16384</code>（共 43054），两个 8192 MiB 切片打包进一张 21.5 GiB 的卡，而不是分散到两张卡。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="3-容器级监控指标正常导出">3. 容器级监控指标正常导出<a href="https://project-hami.io/zh/blog/volcano-ascend-vnpu-soft-slicing#3-%E5%AE%B9%E5%99%A8%E7%BA%A7%E7%9B%91%E6%8E%A7%E6%8C%87%E6%A0%87%E6%AD%A3%E5%B8%B8%E5%AF%BC%E5%87%BA" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>插件（不是业务 Pod）在 <code>:9395</code> 上提供 Prometheus 指标：</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">hami_vgpu_memory_limit_bytes{container="npu",...,pod="ascend-vnpu-check",vdevice_index="0"} 8.589934592e+09</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">hami_vgpu_memory_limit_bytes{container="npu",...,pod="ascend-vnpu-check-2",vdevice_index="0"} 8.589934592e+09</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">hami_host_gpu_memory_used_bytes{device_index="0",device_type="Ascend-Atlas 300I Pro",...} 1.937768448e+09</span><br></div></code></pre></div></div>
<p><code>8.589934592e+09</code> 字节恰好是 8192 MiB，与两个 Pod 的申请值一致，且两个 vdevice 挂在同一张物理卡的 UUID 上。该端点还导出 <code>hami_vgpu_memory_used_bytes</code>、<code>hami_container_device_utilization_ratio</code> 和 <code>hami_host_gpu_utilization_ratio</code>。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="验证结果">验证结果<a href="https://project-hami.io/zh/blog/volcano-ascend-vnpu-soft-slicing#%E9%AA%8C%E8%AF%81%E7%BB%93%E6%9E%9C" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<table><thead><tr><th style="text-align:left">验证项</th><th style="text-align:left">结果</th><th style="text-align:left">证据</th></tr></thead><tbody><tr><td style="text-align:left">Volcano 源码编译（aarch64）</td><td style="text-align:left">通过</td><td style="text-align:left">3 个镜像，scheduler <code>--version</code> 输出 commit <code>7d950432...</code></td></tr><tr><td style="text-align:left">插件镜像携带匹配的 libvnpu</td><td style="text-align:left">通过</td><td style="text-align:left"><code>libvnpu.so</code> 资产已对照匹配驱动 25.5.1 的版本核验</td></tr><tr><td style="text-align:left">Volcano + HAMi 模式 deviceshare</td><td style="text-align:left">通过</td><td style="text-align:left">调度器日志加载 <code>AscendHAMiVNPUEnable: "true"</code></td></tr><tr><td style="text-align:left">节点资源注册</td><td style="text-align:left">通过</td><td style="text-align:left"><code>Ascend310P: 14</code>、<code>Ascend310P-memory: 43054</code></td></tr><tr><td style="text-align:left">显存切片隔离</td><td style="text-align:left">通过</td><td style="text-align:left">容器内 <code>0 / 8192</code>，宿主机 <code>1848 / 21525</code></td></tr><tr><td style="text-align:left">binpack 共卡</td><td style="text-align:left">通过</td><td style="text-align:left">两个 Pod 同为 Bus-Id <code>0000:81:00.0</code>，<code>Global Manager #0/#1</code></td></tr><tr><td style="text-align:left">资源计量</td><td style="text-align:left">通过</td><td style="text-align:left">节点已分配 2 vNPU、16384 MiB</td></tr><tr><td style="text-align:left">监控</td><td style="text-align:left">通过</td><td style="text-align:left"><code>:9395</code> 导出宿主机/容器/vdevice 三层指标</td></tr></tbody></table>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="动手前值得知道的坑">动手前值得知道的坑<a href="https://project-hami.io/zh/blog/volcano-ascend-vnpu-soft-slicing#%E5%8A%A8%E6%89%8B%E5%89%8D%E5%80%BC%E5%BE%97%E7%9F%A5%E9%81%93%E7%9A%84%E5%9D%91" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class=""><strong><code>libvnpu.so</code> 必须与 NPU 驱动匹配。</strong> 不匹配不会报错，容器内 <code>npu-smi</code> 只是永远卡在 <code>Initialize SchedulerClient...</code>。请从与驱动匹配的官方镜像版本拷贝资产并校验 md5。</li>
<li class=""><strong>Docker 与 containerd 的镜像存储是隔离的。</strong> 用 <code>ctr -n k8s.io images import</code> 导入，否则等着收 <code>ErrImageNeverPull</code>。</li>
<li class=""><strong>Helm 里镜像拉取策略的 key 是 <code>basic.image_pull_policy</code></strong>（下划线），不是 <code>scheduler.imagePullPolicy</code>。key 写错时节点会去拉取只有本地才有的镜像。</li>
<li class=""><strong>v1.4.0 不注册 <code>-core</code> 资源。</strong> Pod spec 只需要 <code>huawei.com/Ascend310P</code>（卡数）与 <code>huawei.com/Ascend310P-memory</code>（MiB）；配置里的 <code>resourceCoreName</code> 不会上报为节点资源。</li>
<li class=""><strong>指标在插件 Pod 上。</strong> 在业务 Pod 里 curl <code>:9395</code> 毫无响应；要按 label 选中 DaemonSet Pod。</li>
<li class=""><strong>卸载 Volcano 后 <code>volcano-system</code> 可能卡在 Terminating</strong>（webhook 已删除之后）。清理 namespace 的 finalizer 即可解除。</li>
</ul>
<p>这里的软切分是运行时 API 层的强制（<code>libvnpu.so</code> 软件拦截），不是 SR-IOV 式的硬件安全边界，这一点与 HAMi-core 在 GPU 上的定性相同。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="延伸阅读">延伸阅读<a href="https://project-hami.io/zh/blog/volcano-ascend-vnpu-soft-slicing#%E5%BB%B6%E4%BC%B8%E9%98%85%E8%AF%BB" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class="">完整流程（每条命令与真实输出）：<a class="" href="https://project-hami.io/zh/tutorials/labs/volcano-ascend-vnpu">实验 13：用 Volcano + HAMi-core 软切分昇腾 310P3 vNPU</a></li>
<li class="">用户指南：<a class="" href="https://project-hami.io/zh/docs/installation/how-to-use-volcano-ascend">Volcano 中的华为昇腾设备</a> 与 <a class="" href="https://project-hami.io/zh/docs/userguide/ascend-device/enable-ascend-sharing">启用昇腾共享</a></li>
<li class="">昇腾硬切分：<a href="https://www.hiascend.com/developer/techArticles/20251212-1" target="_blank" rel="noopener noreferrer" class="">《NPU 卡虚拟化硬切分参考实践》（昇腾社区）</a></li>
<li class="">组件仓库：<a href="https://github.com/Project-HAMi/ascend-device-plugin" target="_blank" rel="noopener noreferrer" class="">Project-HAMi/ascend-device-plugin</a> · <a href="https://github.com/Project-HAMi/hami-vnpu-core" target="_blank" rel="noopener noreferrer" class="">Project-HAMi/hami-vnpu-core</a> · <a href="https://github.com/volcano-sh/volcano" target="_blank" rel="noopener noreferrer" class="">volcano-sh/volcano</a></li>
<li class="">相关实验：<a class="" href="https://project-hami.io/zh/tutorials/labs/volcano-vgpu-gang-queue">实验 8：Volcano vGPU、Gang 调度与队列限制</a> 把同一套调度器用于 NVIDIA GPU</li>
</ul>]]></content>
        <author>
            <name>宋净超（Jimmy Song）</name>
            <uri>https://github.com/rootsongjc</uri>
        </author>
        <category label="HAMi" term="HAMi"/>
        <category label="Volcano" term="Volcano"/>
        <category label="Ascend" term="Ascend"/>
        <category label="vNPU" term="vNPU"/>
        <category label="软切分" term="软切分"/>
        <category label="Kubernetes" term="Kubernetes"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[KAI Scheduler 与 HAMi 的 GPU 显存硬隔离：运行原理与实践]]></title>
        <id>https://project-hami.io/zh/blog/kai-scheduler-hami-gpu-memory-hard-isolation</id>
        <link href="https://project-hami.io/zh/blog/kai-scheduler-hami-gpu-memory-hard-isolation"/>
        <updated>2026-08-11T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[简要说明 KAI Scheduler 与 HAMi-core 的隔离链路，并通过可复现的 GKE 实测证明：共享同一张 NVIDIA T4 的两个 Pod 都无法越过各自的显存配额。]]></summary>
        <content type="html"><![CDATA[<p>上一篇 <a class="" href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler">《HAMi-core 被 NVIDIA KAI Scheduler 采用》</a>已经介绍了 KAI Scheduler 和这项集成的协作背景。本文不再重复铺垫，只回答一个问题：<strong>KAI Scheduler 把两个 Pod 调度到同一张 GPU 后，HAMi-core 是否真的能限制每个 Pod 的显存用量？</strong></p>
<p>我们在 GKE 1.35/COS/CDI 上验证了当前文档支持的组合：KAI Scheduler v0.17.0 与 <code>kai-resource-isolator</code> 1.1.0-chart。两个 Pod 共享同一张 NVIDIA T4，各自看到 4147 MiB 上限；申请 3 GiB 成功，累计申请 5 GiB 失败。可选的 monitor 也导出了两个 Pod 的实时显存上限与用量。</p>
<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"></path></svg></span>关于实测输出</div><div class="admonitionContent_BuS1"><p>下文的 UUID、显存上限、CUDA 分配结果和 monitor 指标均来自该次 GKE 实测。资源后缀和地址在不同集群中会变化。</p></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="集成如何工作">集成如何工作<a href="https://project-hami.io/zh/blog/kai-scheduler-hami-gpu-memory-hard-isolation#%E9%9B%86%E6%88%90%E5%A6%82%E4%BD%95%E5%B7%A5%E4%BD%9C" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>整条链路只有三项职责：</p>
<ul>
<li class=""><strong>KAI Scheduler</strong> 决定 Pod 使用哪张 GPU，并通过 <code>CUDA_DEVICE_MEMORY_LIMIT</code> 注入计算出的显存配额。</li>
<li class=""><strong><code>kai-resource-isolator</code></strong> 分发 <code>libvgpu.so</code>；其 webhook 为业务 Pod 挂载该库并配置 <code>ld.so.preload</code>。可选的 monitor 读取节点本地 HAMi 缓存，在 <code>:9394</code> 暴露 <code>hami_*</code> 指标。</li>
<li class=""><strong>HAMi-core（<code>libvgpu.so</code>）</strong> 拦截 <code>cudaMalloc</code> 等 CUDA 调用，拒绝超出配额的显存分配。</li>
</ul>
<figure class="mermaid-figure"><figcaption>KAI Scheduler 与 HAMi-core 的隔离链路</figcaption></figure>
<p><code>CUDA_DEVICE_MEMORY_LIMIT</code> 是调度层与隔离层之间的契约。KAI 不需要知道 CUDA 调用如何被拦截，HAMi-core 也不需要知道配额如何计算。KAI 保留自己的调度逻辑；它集成的是 HAMi-core，而不是用完整的 HAMi 平台替换自身调度器。</p>
<p>容器启动时，动态链接器会先于 CUDA 库加载 <code>libvgpu.so</code>。HAMi-core 读取注入的配额，跟踪容器显存用量，改写设备查询结果，使 <code>nvidia-smi</code> 等工具只显示该配额；新的分配一旦越界就返回错误。这是在 CUDA API 层强制执行，并非依赖应用自觉遵守一个数值。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="在已有-gpu-集群上的标准集成步骤">在已有 GPU 集群上的标准集成步骤<a href="https://project-hami.io/zh/blog/kai-scheduler-hami-gpu-memory-hard-isolation#%E5%9C%A8%E5%B7%B2%E6%9C%89-gpu-%E9%9B%86%E7%BE%A4%E4%B8%8A%E7%9A%84%E6%A0%87%E5%87%86%E9%9B%86%E6%88%90%E6%AD%A5%E9%AA%A4" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>下面默认 Kubernetes 集群中的 NVIDIA GPU 已经可用：节点能够上报 <code>nvidia.com/gpu</code>，普通整卡 Pod 也能正常执行 <code>nvidia-smi</code>。这里只写标准集成路径，不包含实验 12 中针对 GKE 的特殊适配。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="1-安装-kai-scheduler">1. 安装 KAI Scheduler<a href="https://project-hami.io/zh/blog/kai-scheduler-hami-gpu-memory-hard-isolation#1-%E5%AE%89%E8%A3%85-kai-scheduler" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>启用 GPU 共享与 <code>hamicore</code> binder 插件：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm </span><span class="token function" style="color:rgb(130, 170, 255)">install</span><span class="token plain"> kai-scheduler </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--namespace</span><span class="token plain"> kai-scheduler --create-namespace </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--version</span><span class="token plain"> v0.17.0 </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--set</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">global.gpuSharing</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">true </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--set</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">binder.plugins.hamicore.enabled</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">true</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> kai-scheduler </span><span class="token function" style="color:rgb(130, 170, 255)">wait</span><span class="token plain"> </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--for</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">condition</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">available </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--timeout</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">180s deploy </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--all</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> kai-scheduler </span><span class="token function" style="color:rgb(130, 170, 255)">wait</span><span class="token plain"> </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--for</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">condition</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">Ready </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--timeout</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">300s config/kai-config</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl get pods </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> kai-scheduler</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl get queues</span><br></div></code></pre></div></div>
<p>正常情况下，KAI 的所有组件都处于运行状态，并且默认父子队列已经创建。Pod 后缀会因环境而异：</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">NAME                                  READY   STATUS</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">admission-...                         1/1     Running</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">binder-...                            1/1     Running</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kai-operator-...                      1/1     Running</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kai-scheduler-default-...             1/1     Running</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">pod-grouper-...                       1/1     Running</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">podgroup-controller-...               1/1     Running</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">queue-controller-...                  1/1     Running</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">NAME                   PARENT</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">default-parent-queue</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">default-queue          default-parent-queue</span><br></div></code></pre></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="2-安装-kai-resource-isolator">2. 安装 kai-resource-isolator<a href="https://project-hami.io/zh/blog/kai-scheduler-hami-gpu-memory-hard-isolation#2-%E5%AE%89%E8%A3%85-kai-resource-isolator" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>安装 HAMi-core 库分发组件、注入 webhook 和可选的 monitor：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm </span><span class="token function" style="color:rgb(130, 170, 255)">install</span><span class="token plain"> kai-resource-isolator </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  oci://docker.io/projecthami/kai-resource-isolator </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--namespace</span><span class="token plain"> kai-resource-isolator --create-namespace </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--version</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1.1</span><span class="token plain">.0-chart </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--set</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">monitor.enabled</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">true</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl rollout status ds/kai-resource-isolator-libsync </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> kai-resource-isolator </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--timeout</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">300s</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl rollout status ds/kai-resource-isolator-monitor </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> kai-resource-isolator </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--timeout</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">300s</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl rollout status deploy/kai-resource-isolator-webhook </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> kai-resource-isolator </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--timeout</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">300s</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl get pods </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> kai-resource-isolator</span><br></div></code></pre></div></div>
<p>每个 GPU 节点上都应该有一个就绪的 libsync Pod 和一个就绪的 monitor Pod，同时 webhook 也应处于就绪状态：</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">NAME                                  READY   STATUS</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kai-resource-isolator-libsync-...     1/1     Running</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kai-resource-isolator-monitor-...     1/1     Running</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kai-resource-isolator-webhook-...     1/1     Running</span><br></div></code></pre></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="3-运行一个共享-gpu-pod">3. 运行一个共享 GPU Pod<a href="https://project-hami.io/zh/blog/kai-scheduler-hami-gpu-memory-hard-isolation#3-%E8%BF%90%E8%A1%8C%E4%B8%80%E4%B8%AA%E5%85%B1%E4%BA%AB-gpu-pod" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p><code>gpu-memory</code> 注解使用不带单位后缀的整数 MiB。队列标签与 <code>schedulerName</code> 会让 Pod 经 KAI Scheduler 调度：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl apply </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-f</span><span class="token plain"> - </span><span class="token operator" style="color:rgb(137, 221, 255)">&lt;&lt;</span><span class="token string" style="color:rgb(195, 232, 141)">'EOF'</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">apiVersion: v1</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">kind: Pod</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">metadata:</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">  name: kai-hami-check</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">  labels:</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">    kai.scheduler/queue: default-queue</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">  annotations:</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">    gpu-memory: "4096"</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">spec:</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">  schedulerName: kai-scheduler</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">  containers:</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">    - name: cuda</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">      image: nvidia/cuda:12.4.1-base-ubuntu22.04</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">      command: ["sleep", "infinity"]</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">EOF</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl </span><span class="token function" style="color:rgb(130, 170, 255)">wait</span><span class="token plain"> </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--for</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">condition</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">Ready pod/kai-hami-check </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--timeout</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">5m</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl get pod kai-hami-check </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-o</span><span class="token plain"> wide</span><br></div></code></pre></div></div>
<p>Pod 应该处于 <code>Running</code> 状态，并被调度到一个 GPU 节点：</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">NAME             READY   STATUS    NODE</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kai-hami-check   1/1     Running   gpu-node-1</span><br></div></code></pre></div></div>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="4-检查调度层到运行时的交接">4. 检查调度层到运行时的交接<a href="https://project-hami.io/zh/blog/kai-scheduler-hami-gpu-memory-hard-isolation#4-%E6%A3%80%E6%9F%A5%E8%B0%83%E5%BA%A6%E5%B1%82%E5%88%B0%E8%BF%90%E8%A1%8C%E6%97%B6%E7%9A%84%E4%BA%A4%E6%8E%A5" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>检查 KAI 提供的配额、isolator 注入的 preload 文件，以及 HAMi-core 向容器暴露的显存：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">exec</span><span class="token plain"> kai-hami-check -- </span><span class="token function" style="color:rgb(130, 170, 255)">sh</span><span class="token plain"> </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-lc</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">'</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">  test -n "$CUDA_DEVICE_MEMORY_LIMIT"</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">  test -f /usr/local/vgpu/libvgpu.so</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">  printf "limit=%s\n" "$CUDA_DEVICE_MEMORY_LIMIT"</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">  cat /etc/ld.so.preload</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">  nvidia-smi --query-gpu=uuid,memory.total --format=csv,noheader</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">'</span><br></div></code></pre></div></div>
<p>在一张 15360 MiB T4 上申请 4096 MiB 时，实测输出为：</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">limit=4147m</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">/usr/local/vgpu/libvgpu.so</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">GPU-9acc8878-3967-5fb4-c534-43d6fd820fa6, 4147 MiB</span><br></div></code></pre></div></div>
<p>这三行可以证明集成链路已经生效：KAI 提供了显存配额，isolator 注入了 HAMi-core，容器看到的是隔离后的上限而非整卡显存。结果是 4147 而不是刚好 4096 MiB，是因为 KAI 会先把请求换算成两位小数的 GPU fraction，再计算最终强制上限。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="5-检查可选的-monitor">5. 检查可选的 monitor<a href="https://project-hami.io/zh/blog/kai-scheduler-hami-gpu-memory-hard-isolation#5-%E6%A3%80%E6%9F%A5%E5%8F%AF%E9%80%89%E7%9A%84-monitor" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>每个 monitor 只读取本节点上的缓存，因此要选择与业务 Pod 位于同一节点的 monitor。在一个终端中保持下面的端口转发命令运行：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token builtin class-name" style="color:rgb(255, 203, 107)">export</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">WORKLOAD_NODE</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token variable" style="color:rgb(191, 199, 213)">$(</span><span class="token variable" style="color:rgb(191, 199, 213)">kubectl get pod kai-hami-check </span><span class="token variable punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token variable" style="color:rgb(191, 199, 213)"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token variable" style="color:rgb(191, 199, 213)">  </span><span class="token variable parameter variable" style="color:rgb(191, 199, 213)">-o</span><span class="token variable" style="color:rgb(191, 199, 213)"> </span><span class="token variable assign-left variable" style="color:rgb(191, 199, 213)">jsonpath</span><span class="token variable operator" style="color:rgb(137, 221, 255)">=</span><span class="token variable string" style="color:rgb(195, 232, 141)">'{.spec.nodeName}'</span><span class="token variable" style="color:rgb(191, 199, 213)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">export</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">MONITOR_POD</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token variable" style="color:rgb(191, 199, 213)">$(</span><span class="token variable" style="color:rgb(191, 199, 213)">kubectl get pods </span><span class="token variable parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token variable" style="color:rgb(191, 199, 213)"> kai-resource-isolator </span><span class="token variable punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token variable" style="color:rgb(191, 199, 213)"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token variable" style="color:rgb(191, 199, 213)">  </span><span class="token variable parameter variable" style="color:rgb(191, 199, 213)">-l</span><span class="token variable" style="color:rgb(191, 199, 213)"> app.kubernetes.io/component</span><span class="token variable operator" style="color:rgb(137, 221, 255)">=</span><span class="token variable" style="color:rgb(191, 199, 213)">kai-vgpu-monitor </span><span class="token variable punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token variable" style="color:rgb(191, 199, 213)"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token variable" style="color:rgb(191, 199, 213)">  --field-selector</span><span class="token variable operator" style="color:rgb(137, 221, 255)">=</span><span class="token variable string" style="color:rgb(195, 232, 141)">"spec.nodeName=</span><span class="token variable string variable" style="color:rgb(191, 199, 213)">$WORKLOAD_NODE</span><span class="token variable string" style="color:rgb(195, 232, 141)">"</span><span class="token variable" style="color:rgb(191, 199, 213)"> </span><span class="token variable punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token variable" style="color:rgb(191, 199, 213)"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token variable" style="color:rgb(191, 199, 213)">  </span><span class="token variable parameter variable" style="color:rgb(191, 199, 213)">-o</span><span class="token variable" style="color:rgb(191, 199, 213)"> </span><span class="token variable assign-left variable" style="color:rgb(191, 199, 213)">jsonpath</span><span class="token variable operator" style="color:rgb(137, 221, 255)">=</span><span class="token variable string" style="color:rgb(195, 232, 141)">'{range .items[*]}{.metadata.name}{"\n"}{end}'</span><span class="token variable" style="color:rgb(191, 199, 213)"> </span><span class="token variable operator" style="color:rgb(137, 221, 255)">|</span><span class="token variable" style="color:rgb(191, 199, 213)"> </span><span class="token variable function" style="color:rgb(130, 170, 255)">head</span><span class="token variable" style="color:rgb(191, 199, 213)"> </span><span class="token variable parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token variable" style="color:rgb(191, 199, 213)"> </span><span class="token variable number" style="color:rgb(247, 140, 108)">1</span><span class="token variable" style="color:rgb(191, 199, 213)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">test</span><span class="token plain"> </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"</span><span class="token string variable" style="color:rgb(191, 199, 213)">$MONITOR_POD</span><span class="token string" style="color:rgb(195, 232, 141)">"</span><span class="token plain"> </span><span class="token operator" style="color:rgb(137, 221, 255)">||</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">{</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">echo</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"</span><span class="token string variable" style="color:rgb(191, 199, 213)">$WORKLOAD_NODE</span><span class="token string" style="color:rgb(195, 232, 141)"> 上没有运行 monitor Pod"</span><span class="token plain"> </span><span class="token operator" style="color:rgb(137, 221, 255)">&gt;</span><span class="token file-descriptor important">&amp;2</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token builtin class-name" style="color:rgb(255, 203, 107)">exit</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token punctuation" style="color:rgb(199, 146, 234)">}</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl port-forward </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> kai-resource-isolator </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token string" style="color:rgb(195, 232, 141)">"pod/</span><span class="token string variable" style="color:rgb(191, 199, 213)">$MONITOR_POD</span><span class="token string" style="color:rgb(195, 232, 141)">"</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">9394</span><span class="token plain">:9394</span><br></div></code></pre></div></div>
<p>在另一个终端确认端点中存在该 Pod 的指标：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token function" style="color:rgb(130, 170, 255)">curl</span><span class="token plain"> </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-s</span><span class="token plain"> http://127.0.0.1:9394/metrics </span><span class="token operator" style="color:rgb(137, 221, 255)">|</span><span class="token plain"> </span><span class="token function" style="color:rgb(130, 170, 255)">grep</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token string" style="color:rgb(195, 232, 141)">'hami_vgpu_memory_limit_bytes.*pod="kai-hami-check"'</span><br></div></code></pre></div></div>
<p>输出格式应类似：</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">hami_vgpu_memory_limit_bytes{...,pod="kai-hami-check",...} 4.348444672e+09</span><br></div></code></pre></div></div>
<p>以上检查足以确认基本集成链路正确。要证明显示出来的上限确实无法越过，还需要执行一次 CUDA 分配测试：配额内成功、超额失败。下一节汇总了这项实测结果，实验 12 则提供完整程序和操作过程。</p>
<p>完成基本检查后删除测试 Pod：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">kubectl delete pod kai-hami-check</span><br></div></code></pre></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="gke-实测隔离是否真的生效">GKE 实测：隔离是否真的生效？<a href="https://project-hami.io/zh/blog/kai-scheduler-hami-gpu-memory-hard-isolation#gke-%E5%AE%9E%E6%B5%8B%E9%9A%94%E7%A6%BB%E6%98%AF%E5%90%A6%E7%9C%9F%E7%9A%84%E7%94%9F%E6%95%88" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>验证环境是 GKE 1.35/COS/CDI 集群，包含三个 <code>n1-standard-2</code> 节点，每个节点有一张 NVIDIA T4。KAI Scheduler v0.17.0 负责共享调度，<code>kai-resource-isolator</code> 1.1.0-chart 负责注入 HAMi-core。</p>
<table><thead><tr><th style="text-align:left">检查项</th><th style="text-align:left">实测结果</th><th style="text-align:left">证明了什么</th></tr></thead><tbody><tr><td style="text-align:left">节点与 GPU UUID</td><td style="text-align:left">两个 Pod 位于同一单卡节点，并返回 <code>GPU-9acc8878-...</code></td><td style="text-align:left">它们共享同一张物理 T4</td></tr><tr><td style="text-align:left">可见显存</td><td style="text-align:left">两个 Pod 均报告 <code>4147 MiB</code>，整卡报告 <code>15360 MiB</code></td><td style="text-align:left">HAMi-core 暴露了 KAI 分配给各 Pod 的配额</td></tr><tr><td style="text-align:left">CUDA 分配</td><td style="text-align:left">3 GiB 成功，累计 5 GiB 返回 <code>out of memory</code></td><td style="text-align:left">上限确实执行，而不只是显示值变化</td></tr><tr><td style="text-align:left">并发隔离</td><td style="text-align:left">Pod A 持有 3 GiB 时，Pod B 仍能申请自己的 3 GiB</td><td style="text-align:left">一个 Pod 无法占用另一个 Pod 的配额</td></tr><tr><td style="text-align:left">Monitor 指标</td><td style="text-align:left">同节点 <code>:9394/metrics</code> 返回两个 Pod 的 4.348 GB 上限和 3.328 GB 实时用量</td><td style="text-align:left">monitor 从容器缓存导出了非空的 Pod 级指标</td></tr></tbody></table>
<p>超额分配时，HAMi-core 记录：</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">Device 0 OOM 5475663872 / 4348444672</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">allocate another 2 GiB: out of memory</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">PASS: in-quota allocation succeeded and over-quota allocation failed</span><br></div></code></pre></div></div>
<p>这些结果把“调度到同一张卡”“容器内只见自身配额”“CUDA 分配确实越不过上限”和“monitor 能观测实时用量”连成了一条完整证据链。由于 monitor 读取节点本地缓存，实验会直接查询业务 Pod 所在节点的 monitor 实例，避免 Service 把请求转发到其他节点。</p>
<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 14 16"><path fill-rule="evenodd" d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"></path></svg></span>隔离边界</div><div class="admonitionContent_BuS1"><p>这里验证的是 CUDA API 层的显存限制，不是 MIG 一类硬件安全边界。本次 GKE 兼容路径还使用了特权业务容器，因此不应作为不可信多租户安全方案。</p></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="复现实测结果">复现实测结果<a href="https://project-hami.io/zh/blog/kai-scheduler-hami-gpu-memory-hard-isolation#%E5%A4%8D%E7%8E%B0%E5%AE%9E%E6%B5%8B%E7%BB%93%E6%9E%9C" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>标准 KAI + HAMi-core 安装链路并不长。GKE 1.35/COS/CDI 还需要针对只读根文件系统、RuntimeClass、NVML 库路径、CDI 设备注入和 PriorityClass 做环境适配。完整操作、实测命令输出与故障排查统一维护在：</p>
<p><strong><a class="" href="https://project-hami.io/zh/tutorials/labs/kai-scheduler-hami-gke">实验 12：在 GKE 上验证 KAI Scheduler 与 HAMi 显存隔离</a></strong></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="下一步">下一步<a href="https://project-hami.io/zh/blog/kai-scheduler-hami-gpu-memory-hard-isolation#%E4%B8%8B%E4%B8%80%E6%AD%A5" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class="">背景与协作过程：<a class="" href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler">《HAMi-core 被 NVIDIA KAI Scheduler 采用》</a></li>
<li class="">用户文档：<a class="" href="https://project-hami.io/zh/docs/next/userguide/kai-scheduler/how-to-use-kai-scheduler">如何在 KAI Scheduler 中使用 HAMi</a></li>
<li class="">相关仓库：<a href="https://github.com/Project-HAMi/KAI-resource-isolator" target="_blank" rel="noopener noreferrer" class="">KAI-resource-isolator</a>、<a href="https://github.com/Project-HAMi/HAMi-core" target="_blank" rel="noopener noreferrer" class="">HAMi-core</a> 和 <a href="https://github.com/kai-scheduler/KAI-Scheduler" target="_blank" rel="noopener noreferrer" class="">KAI-Scheduler</a></li>
</ul>]]></content>
        <author>
            <name>宋净超（Jimmy Song）</name>
            <uri>https://github.com/rootsongjc</uri>
        </author>
        <category label="HAMi" term="HAMi"/>
        <category label="KAI Scheduler" term="KAI Scheduler"/>
        <category label="硬隔离" term="硬隔离"/>
        <category label="GPU 共享" term="GPU 共享"/>
        <category label="Kubernetes" term="Kubernetes"/>
        <category label="云原生" term="云原生"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[LFX Mentorship 2026 Term 3 开放申请：HAMi 四大开源课题等你挑战]]></title>
        <id>https://project-hami.io/zh/blog/lfx-mentorship-2026-term-3</id>
        <link href="https://project-hami.io/zh/blog/lfx-mentorship-2026-term-3"/>
        <updated>2026-08-03T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[HAMi 参与 Linux Foundation LFX Mentorship 2026 第三期（9 月至 11 月），mentee 申请于 8 月 3 日至 8 月 18 日开放。一文了解四大课题、导师团队、时间线与申请流程。]]></summary>
        <content type="html"><![CDATA[<p>Linux Foundation <strong>LFX Mentorship Program</strong> 2026 第三期（Term 3）正式启动，<strong>HAMi</strong> 将在 9 月至 11 月期间指导四个开源课题。</p>
<p>mentee 申请通道于 <strong>2026 年 8 月 3 日</strong>开启，<strong>8 月 18 日</strong>截止。无论你钟情底层 C/C++ 性能优化、GPU 可观测性、容器隔离安全，还是开发者教育与文档，都能找到适合你的方向。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="什么是-lfx-mentorship-program">什么是 LFX Mentorship Program？<a href="https://project-hami.io/zh/blog/lfx-mentorship-2026-term-3#%E4%BB%80%E4%B9%88%E6%98%AF-lfx-mentorship-program" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>LFX Mentorship 是 Linux Foundation 主办的开源导师计划。在为期三个月的周期里，入选的贡献者将与开源项目的维护者（maintainer）一对一协作，完成一个明确的项目。入选的 mentee 可获得由 LFX Mentorship Program 发放的津贴（性质为教育资助，金额核定与发放均由主办方负责，HAMi 社区不承担津贴支付，详见下文 FAQ），并积累真实的生产级经验，在 CNCF 社区留下公开的贡献记录。</p>
<p>HAMi 在本期（Term 3）提供的每个课题大约对应 <strong>350 小时</strong>的工作量，分布在约 12 周的指导期内。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="为什么选择-hami">为什么选择 HAMi？<a href="https://project-hami.io/zh/blog/lfx-mentorship-2026-term-3#%E4%B8%BA%E4%BB%80%E4%B9%88%E9%80%89%E6%8B%A9-hami" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>HAMi 是 CNCF 孵化（Incubating）项目，也是 Kubernetes 的异构算力虚拟化与调度中间件。它不改驱动、不改应用，通过 CUDA API 劫持（HAMi-Core，即注入容器内的 <code>libvgpu.so</code>）在软件层实现 GPU 虚拟化，精细切分 GPU 的显存与算力，让多个工作负载安全共享同一张加速卡。目前已被数百家组织在十多种加速卡上投入生产使用，与 Kubernetes 默认调度器、Volcano、Kueue、Koordinator 以及 NVIDIA KAI Scheduler 完成集成，并由全球近 500 位贡献者共同建设。</p>
<p>加入 HAMi，意味着站在 AI 基础设施浪潮的中心做开源。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="第三期term-3四大课题">第三期（Term 3）四大课题<a href="https://project-hami.io/zh/blog/lfx-mentorship-2026-term-3#%E7%AC%AC%E4%B8%89%E6%9C%9Fterm-3%E5%9B%9B%E5%A4%A7%E8%AF%BE%E9%A2%98" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="1-hami-gpu-共享实战-workshop-与文档">1. HAMi GPU 共享实战 Workshop 与文档<a href="https://project-hami.io/zh/blog/lfx-mentorship-2026-term-3#1-hami-gpu-%E5%85%B1%E4%BA%AB%E5%AE%9E%E6%88%98-workshop-%E4%B8%8E%E6%96%87%E6%A1%A3" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<ul>
<li class="">上游 Issue：<a href="https://github.com/Project-HAMi/website/issues/656" target="_blank" rel="noopener noreferrer" class="">Project-HAMi/website#656</a></li>
<li class="">LFX 申请页：<a href="https://mentorship.lfx.linuxfoundation.org/project/ab5693b7-9759-48e2-b609-f48af6b82206" target="_blank" rel="noopener noreferrer" class="">HAMi GPU Sharing Workshop and Documentation</a></li>
<li class="">技术栈：Kubernetes、HAMi、Markdown、Docusaurus</li>
<li class="">技能要求：技术写作、文档可复现性测试、故障排查、GitHub 工作流</li>
<li class="">导师：Reza Jelveh（<a href="https://github.com/fishman" target="_blank" rel="noopener noreferrer" class="">@fishman</a>）、宋净超（Jimmy Song）（<a href="https://github.com/rootsongjc" target="_blank" rel="noopener noreferrer" class="">@rootsongjc</a>）</li>
</ul>
<p>本项目旨在产出可复现的动手实验（hands-on labs），帮助用户与新贡献者部署 HAMi、理解 GPU 共享，并排查安装、调度与分配中的常见问题。预期成果包括：HAMi GPU 共享实战实验、vLLM/Ray/SGLang 等 AI 推理工作负载示例、常见问题排查指南、带验证步骤的测试用 manifest，以及至少一个面向 HAMi 网站（或导师认可的 HAMi 仓库）的贡献。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="2-修复子进程与-ssh-进程的-gpu-显存隔离">2. 修复子进程与 SSH 进程的 GPU 显存隔离<a href="https://project-hami.io/zh/blog/lfx-mentorship-2026-term-3#2-%E4%BF%AE%E5%A4%8D%E5%AD%90%E8%BF%9B%E7%A8%8B%E4%B8%8E-ssh-%E8%BF%9B%E7%A8%8B%E7%9A%84-gpu-%E6%98%BE%E5%AD%98%E9%9A%94%E7%A6%BB" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<ul>
<li class="">上游 Issue：<a href="https://github.com/Project-HAMi/HAMi/issues/2125" target="_blank" rel="noopener noreferrer" class="">Project-HAMi/HAMi#2125</a></li>
<li class="">LFX 申请页：<a href="https://mentorship.lfx.linuxfoundation.org/project/e5e55e8c-fab1-4453-9d49-a4a5e013c4c5" target="_blank" rel="noopener noreferrer" class="">Fix GPU Memory Isolation for Child and SSH Processes</a></li>
<li class="">技术栈：Kubernetes、Linux 容器、NVIDIA GPU、HAMi-core</li>
<li class="">技能要求：Linux 调试、容器运行时分析、安全推理、C/C++、测试</li>
<li class="">导师：李孟轩（Mengxuan Li）（<a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">@archlitchi</a>）、宋净超（Jimmy Song）（<a href="https://github.com/rootsongjc" target="_blank" rel="noopener noreferrer" class="">@rootsongjc</a>）</li>
</ul>
<p>在 HAMi 管理的容器中，后启动的进程（包括子进程，以及通过 SSH 或新登录 shell 启动的进程）可能无法保持预期的显存上限。本课题将定位并修复这一隔离边界问题。预期成果包括：可复现的测试用例与根因分析、经维护者评审的隔离设计（覆盖隔离边界、兼容性与失败模式）、维护者认可的实现（或在平台受限时经过验证的设计或原型）、覆盖原始进程、子进程、SSH 与新 shell 场景的回归测试，以及配置说明与迁移文档。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="3-降低-hami-core-初始化的锁竞争">3. 降低 HAMi-core 初始化的锁竞争<a href="https://project-hami.io/zh/blog/lfx-mentorship-2026-term-3#3-%E9%99%8D%E4%BD%8E-hami-core-%E5%88%9D%E5%A7%8B%E5%8C%96%E7%9A%84%E9%94%81%E7%AB%9E%E4%BA%89" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<ul>
<li class="">上游 Issue：<a href="https://github.com/Project-HAMi/HAMi/issues/1662" target="_blank" rel="noopener noreferrer" class="">Project-HAMi/HAMi#1662</a></li>
<li class="">LFX 申请页：<a href="https://mentorship.lfx.linuxfoundation.org/project/ce2ebae3-6936-409d-a9d7-c98b109ec814" target="_blank" rel="noopener noreferrer" class="">Reduce HAMi-core Initialization Lock Contention</a></li>
<li class="">技术栈：C/C++、Linux、CUDA、NVML、HAMi-core</li>
<li class="">技能要求：Linux 并发、性能剖析、基准测试（benchmarking）、同步设计、测试</li>
<li class="">导师：李孟轩（Mengxuan Li）（<a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">@archlitchi</a>）、杨守仁（Shouren Yang）（<a href="https://github.com/Shouren" target="_blank" rel="noopener noreferrer" class="">@Shouren</a>）</li>
</ul>
<p>当数百个进程并发初始化 CUDA 并争用同一把共享统一锁时，HAMi-core 的启动性能会明显下降。本课题要在保证正确性的前提下，降低初始化延迟与锁竞争。预期成果包括：并发初始化的可复现基准、经维护者认可的加锁设计、带并发与回归测试的实现、启动延迟、吞吐与资源占用的性能测量，以及设计与运维影响文档。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="4-hami-gpu-可观测性指标与仪表盘">4. HAMi GPU 可观测性：指标与仪表盘<a href="https://project-hami.io/zh/blog/lfx-mentorship-2026-term-3#4-hami-gpu-%E5%8F%AF%E8%A7%82%E6%B5%8B%E6%80%A7%E6%8C%87%E6%A0%87%E4%B8%8E%E4%BB%AA%E8%A1%A8%E7%9B%98" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<ul>
<li class="">上游 Issue：<a href="https://github.com/Project-HAMi/HAMi/issues/2126" target="_blank" rel="noopener noreferrer" class="">Project-HAMi/HAMi#2126</a></li>
<li class="">LFX 申请页：<a href="https://mentorship.lfx.linuxfoundation.org/project/dd302799-03ec-4184-b289-4d59a41fe7ed" target="_blank" rel="noopener noreferrer" class="">HAMi GPU Observability: Metrics and Dashboards</a></li>
<li class="">技术栈：Prometheus、Grafana、OpenTelemetry、Kubernetes、Go、HAMi</li>
<li class="">技能要求：指标埋点、PromQL、仪表盘设计、可观测性、Kubernetes 调试、HAMi 内部原理</li>
<li class="">导师：Mesut Oezdil（<a href="https://github.com/mesutoezdil" target="_blank" rel="noopener noreferrer" class="">@mesutoezdil</a>）、Reza Jelveh（<a href="https://github.com/fishman" target="_blank" rel="noopener noreferrer" class="">@fishman</a>）、宋净超（Jimmy Song）（<a href="https://github.com/rootsongjc" target="_blank" rel="noopener noreferrer" class="">@rootsongjc</a>）</li>
</ul>
<p>本课题面向运维共享 GPU 工作负载的同学，目标是让 HAMi 的可观测性更实用、更一致、文档更完善。重点在实用的指标与仪表盘，同时把链路追踪（tracing）限定在一条导师认可的控制面链路的设计与概念验证上。预期成果包括：现有指标、导出器、标签、仪表盘与文档的差距分析、经维护者评审的 Prometheus 指标与标签改进、一个带版本管理的 Grafana 仪表盘、指标语义与单位及示例 PromQL 文档、一条选定工作流的 OpenTelemetry 链路设计与小范围 POC、基于交付遥测数据的告警或 SLO 指南，以及至少一个涉及指标、仪表盘、测试或文档的合并贡献。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="导师团队">导师团队<a href="https://project-hami.io/zh/blog/lfx-mentorship-2026-term-3#%E5%AF%BC%E5%B8%88%E5%9B%A2%E9%98%9F" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>四个课题由 HAMi 的五位维护者共同指导：</p>
<ul>
<li class=""><strong>李孟轩（Mengxuan Li）</strong>（<a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">@archlitchi</a>）</li>
<li class=""><strong>宋净超（Jimmy Song）</strong>（<a href="https://github.com/rootsongjc" target="_blank" rel="noopener noreferrer" class="">@rootsongjc</a>）</li>
<li class=""><strong>杨守仁（Shouren Yang）</strong>（<a href="https://github.com/Shouren" target="_blank" rel="noopener noreferrer" class="">@Shouren</a>）</li>
<li class=""><strong>Mesut Oezdil</strong>（<a href="https://github.com/mesutoezdil" target="_blank" rel="noopener noreferrer" class="">@mesutoezdil</a>）</li>
<li class=""><strong>Reza Jelveh</strong>（<a href="https://github.com/fishman" target="_blank" rel="noopener noreferrer" class="">@fishman</a>）</li>
</ul>
<p>每个课题至少配备一位技术导师与一位社区导师，分别负责架构与代码评审，以及 onboarding 与 CNCF 工作流指导。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="时间线">时间线<a href="https://project-hami.io/zh/blog/lfx-mentorship-2026-term-3#%E6%97%B6%E9%97%B4%E7%BA%BF" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<table><thead><tr><th>阶段</th><th>时间</th></tr></thead><tbody><tr><td>课题征集（已完成）</td><td>2026 年 7 月 1 日 至 7 月 28 日</td></tr><tr><td><strong>mentee 申请</strong></td><td><strong>2026 年 8 月 3 日 至 8 月 18 日</strong></td></tr><tr><td>申请评审</td><td>2026 年 8 月 19 日 至 9 月 1 日</td></tr><tr><td><strong>导师指导期</strong></td><td><strong>2026 年 9 月 7 日 至 11 月 27 日</strong></td></tr></tbody></table>
<p>约 12 周的指导期大致分为三个阶段：</p>
<ul>
<li class=""><strong>第 1 个月（9 月）：onboarding。</strong> 理解 HAMi 架构与社区流程，搭建本地环境，提交第一个 issue 并争取尽快合并。</li>
<li class=""><strong>第 2 个月（10 月）：主开发。</strong> 推进核心实现，参与社区讨论，在社区会议或 Slack 上同步进展。</li>
<li class=""><strong>第 3 个月（11 月）：交付。</strong> 完成交付物，补充文档，向社区做最终成果展示。</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="如何申请">如何申请<a href="https://project-hami.io/zh/blog/lfx-mentorship-2026-term-3#%E5%A6%82%E4%BD%95%E7%94%B3%E8%AF%B7" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>申请通过 LFX 平台完成。需要准备的材料（具体以各课题 LFX 申请页为准）：</p>
<ol>
<li class="">简历（Resume）</li>
<li class="">求职信（Cover Letter）</li>
<li class="">在读证明或同等材料（School Enrollment Verification）</li>
<li class="">学校或雇主的参与许可（Participation Permission）</li>
<li class="">编程挑战（Coding Challenge）</li>
<li class="">课题特定的自定义前置任务（需提交文件）</li>
</ol>
<p>建议步骤：</p>
<ol>
<li class="">在 <a href="https://mentorship.lfx.linuxfoundation.org/" target="_blank" rel="noopener noreferrer" class="">LFX Mentorship 平台</a> 注册账号。</li>
<li class="">浏览上方四个课题，选择最匹配你技能与兴趣的一个，进入对应的 LFX 申请页。</li>
<li class="">在申请截止（<strong>8 月 18 日</strong>）前提交全部材料并完成 coding challenge。</li>
<li class="">等待评审结果（<strong>8 月 19 日 至 9 月 1 日</strong>）。</li>
</ol>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="常见问题">常见问题<a href="https://project-hami.io/zh/blog/lfx-mentorship-2026-term-3#%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><strong>我可以同时申请多个课题吗？</strong> 建议聚焦你最感兴趣、最匹配的一个，把精力投入到高质量的申请材料上。</p>
<p><strong>不是在校学生可以申请吗？</strong> 可以。LFX 面向广泛的贡献者，但需要提供在读证明或雇主的参与许可，具体以平台要求为准。</p>
<p><strong>津贴是多少？按 PPP 计算。</strong> 津贴由 LFX Mentorship Program（Linux Foundation 主办，Linux Foundation、Intel 等提供资金支持）发放，性质为教育资助（educational grant），并非工资报酬。金额按购买力平价（PPP）计算，范围为 1,000 至 6,600 美元，具体取决于 mentee 所在的国家或地区，分两期在中期评估（10 月 21 日）与最终评估（11 月 25 日）之后发放。津贴的核定与发放均由主办方负责，HAMi 社区不承担津贴支付，具体规则以 <a href="https://www.linuxfoundation.org/blog/strengthening-lfx-mentoring-a-fairer-stipend-model-and-a-shared-standard-of-excellence" target="_blank" rel="noopener noreferrer" class="">Linux Foundation 官方说明</a> 为准。</p>
<p><strong>我没有 HAMi 贡献经验，可以申请吗？</strong> 可以。提前在 <a href="https://github.com/Project-HAMi/HAMi" target="_blank" rel="noopener noreferrer" class="">HAMi GitHub</a> 熟悉代码、尝试解决一个 good-first-issue，会让你的申请更有竞争力。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="立即开始">立即开始<a href="https://project-hami.io/zh/blog/lfx-mentorship-2026-term-3#%E7%AB%8B%E5%8D%B3%E5%BC%80%E5%A7%8B" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>申请截止日期是 <strong>2026 年 8 月 18 日</strong>，时间紧迫。现在就去选择你的课题：</p>
<ul>
<li class="">四个课题的 LFX 申请页见上文各章节</li>
<li class="">HAMi 代码仓库：<a href="https://github.com/Project-HAMi/HAMi" target="_blank" rel="noopener noreferrer" class="">Project-HAMi/HAMi</a></li>
<li class="">加入 HAMi Discord：<a href="https://discord.gg/Amhy7XmbNq" target="_blank" rel="noopener noreferrer" class="">discord.gg/Amhy7XmbNq</a></li>
<li class="">更多社区入口：<a class="" href="https://project-hami.io/zh/community">HAMi 社区</a></li>
</ul>
<p>期待在 Term 3 与你一起做开源。</p>]]></content>
        <author>
            <name>HAMi 社区</name>
        </author>
        <category label="LFX" term="LFX"/>
        <category label="Mentorship" term="Mentorship"/>
        <category label="CNCF" term="CNCF"/>
        <category label="社区" term="社区"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[HAMi 亮相 KubeCon + CloudNativeCon 日本 2026]]></title>
        <id>https://project-hami.io/zh/blog/kubecon-japan-2026-recap</id>
        <link href="https://project-hami.io/zh/blog/kubecon-japan-2026-recap"/>
        <updated>2026-08-01T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[回顾 HAMi 在横滨 KubeCon + CloudNativeCon 日本 2026 的亮相：一场关于 SNOW 千卡生产蓝图的主舞台分享，以及 HAMi 展台。]]></summary>
        <content type="html"><![CDATA[<p><a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-japan/" target="_blank" rel="noopener noreferrer" class="">KubeCon + CloudNativeCon 日本 2026</a> 于 7 月 28 至 30 日在 Pacifico Yokohama 举办。HAMi 以一场主舞台分享和 Project Pavilion 的展台参与本届大会。<a href="https://github.com/fishman" target="_blank" rel="noopener noreferrer" class="">Reza Jelveh</a> 登台分享，并在展台值守。</p>
<p>完整的回顾、演讲幻灯片与 SNOW 案例，请见 <a class="" href="https://project-hami.io/zh/landing/kubecon-japan">KubeCon 日本 2026 专页</a>。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="keynote-中的-hami">Keynote 中的 HAMi<a href="https://project-hami.io/zh/blog/kubecon-japan-2026-recap#keynote-%E4%B8%AD%E7%9A%84-hami" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="HAMi 出现在 Chris Aniszczyk 与 Jonathan Bryce 在 KubeCon 日本 2026 的 Keynote 幻灯片中" src="https://project-hami.io/zh/assets/images/keynote-cb04067d92ec21cd840aded6be7ce4fb.jpg" width="1702" height="1276" class="img_ev3q"></p>
<p>HAMi 出现在开场 Keynote 中，CNCF 首席技术官 Chris Aniszczyk 与 Linux 基金会云与基础设施执行董事 Jonathan Bryce 的幻灯片对其进行了介绍。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="主舞台共享-gpu-调度与主动自动伸缩">主舞台：共享 GPU 调度与主动自动伸缩<a href="https://project-hami.io/zh/blog/kubecon-japan-2026-recap#%E4%B8%BB%E8%88%9E%E5%8F%B0%E5%85%B1%E4%BA%AB-gpu-%E8%B0%83%E5%BA%A6%E4%B8%8E%E4%B8%BB%E5%8A%A8%E8%87%AA%E5%8A%A8%E4%BC%B8%E7%BC%A9" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="Reza Jelveh（密瓜智能）与 Jeonghyun Kim（SNOW）在 KubeCon 日本 2026 的分享" src="https://project-hami.io/zh/assets/images/reza-snow-e9d17e858119267f1ed35e97281417ce.jpg" width="2048" height="1536" class="img_ev3q"></p>
<p><a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-japan/program/schedule/?id=1182713" target="_blank" rel="noopener noreferrer" class=""><strong>Shared GPU Scheduling &amp; Proactive Autoscaling: A Production Blueprint for 1000+ GPUs</strong></a> 这场分享，由 Jeonghyun Kim（SNOW AI 工程师）与 Reza Jelveh 共同呈现。</p>
<p>该分享介绍了 SNOW 如何以超过 1000 张 NVIDIA A100 GPU，服务 Snow、Epik、B612 三款覆盖超过 2 亿用户的 AI 应用。其架构以 HAMi 完成 GPU 共享，以 KEDA 实现主动式扩容，在相同流量下将 GPU 需求减半，并提升了负载波动下的恢复能力。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="展台">展台<a href="https://project-hami.io/zh/blog/kubecon-japan-2026-recap#%E5%B1%95%E5%8F%B0" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="KubeCon 日本 2026 Project Pavilion 的 HAMi 展台" src="https://project-hami.io/zh/assets/images/hami-booth-c50e374778fe13eb2f3e576044cce293.jpg" width="1080" height="1440" class="img_ev3q"></p>
<p>在 Project Pavilion 的 HAMi 展台，前来交流的工程师主要关注以下问题：</p>
<ul>
<li class="">如何将 HAMi 引入现有集群</li>
<li class="">HAMi 与 <strong>Volcano</strong>、<strong>Kueue</strong>、<strong>NVIDIA KAI Scheduler</strong> 如何协同</li>
<li class="">实际的 GPU 调度难题：显存碎片化、批处理任务导致的 GPU 闲置，以及推理工作负载难以获得稳定的 GPU 切片</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="进一步了解">进一步了解<a href="https://project-hami.io/zh/blog/kubecon-japan-2026-recap#%E8%BF%9B%E4%B8%80%E6%AD%A5%E4%BA%86%E8%A7%A3" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>本次活动的演讲、SNOW 案例、完整指标、幻灯片与参与方式，均可在 <a class="" href="https://project-hami.io/zh/landing/kubecon-japan">KubeCon 日本 2026 专页</a> 查看。如需参与讨论，欢迎加入 <a class="" href="https://project-hami.io/zh/community">HAMi 社区</a>。</p>]]></content>
        <author>
            <name>HAMi 社区</name>
        </author>
        <category label="KubeCon" term="KubeCon"/>
        <category label="GPU" term="GPU"/>
        <category label="Kubernetes" term="Kubernetes"/>
        <category label="AI" term="AI"/>
        <category label="Japan" term="Japan"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[「你的算力用的好么？」李孟轩 vLLM Meetup 分享回顾：vLLM 推理集群优化的三个阶段]]></title>
        <id>https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap</id>
        <link href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap"/>
        <updated>2026-07-16T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[2026 年 7 月 16 日，密瓜智能联合创始人兼 CTO、HAMi 作者李孟轩在 vLLM Meetup 上分享了 vLLM 推理集群优化的三个阶段：从单机进程、Kubernetes 工作负载、PD 分离，到结合 HAMi GPU 虚拟化的极致算力优化。本文为完整图文回顾。]]></summary>
        <content type="html"><![CDATA[<p><img decoding="async" loading="lazy" alt="vLLM 推理集群优化的三个阶段｜李孟轩" src="https://project-hami.io/zh/assets/images/title-e98a714cde4dcbe87bc1d09455724d93.webp" width="2000" height="1125" class="img_ev3q"></p>
<p>2026 年 7 月 16 日，密瓜智能（Dynamia）联合创始人兼 CTO、HAMi 作者 <strong>李孟轩</strong> 在 vLLM Meetup 上做了一场关于 <strong>vLLM 部署与算力优化</strong> 的技术分享。围绕一个直击痛点的问题：<strong>「你的算力用的好么？」</strong>，他系统梳理了 vLLM 推理集群从"能跑起来"到"把算力榨干"的演进路径，把整个优化过程拆解为清晰的三个阶段。</p>
<p>本文结合分享 PPT 与现场纪要，带大家完整回顾这场干货满满的分享。</p>
<p><img decoding="async" loading="lazy" alt="李孟轩在 vLLM Meetup 现场分享" src="https://project-hami.io/zh/assets/images/li-mengxuan-speaking-4c4926ccde666e51e8044897967ed761.webp" width="1600" height="1067" class="img_ev3q"></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="视频回放">视频回放<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#%E8%A7%86%E9%A2%91%E5%9B%9E%E6%94%BE" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<div style="position:relative;padding-bottom:56.25%;height:0;overflow:hidden;border-radius:12px"><iframe src="https://player.bilibili.com/player.html?bvid=BV1amK462EhY&amp;page=1&amp;high_quality=1&amp;danmaku=0" title="你的算力用的好么？vLLM Meetup上海站，李孟轩" scrolling="no" border="0" frameborder="no" framespacing="0" style="position:absolute;top:0;left:0;width:100%;height:100%;border:0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen=""></iframe></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="核心问题你的算力真的用好了吗">核心问题：你的算力真的用好了吗？<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#%E6%A0%B8%E5%BF%83%E9%97%AE%E9%A2%98%E4%BD%A0%E7%9A%84%E7%AE%97%E5%8A%9B%E7%9C%9F%E7%9A%84%E7%94%A8%E5%A5%BD%E4%BA%86%E5%90%97" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>大模型推理成本居高不下，GPU 越来越贵。但在生产环境里，很多团队的推理集群并没有把昂贵的算力用满。李孟轩用一个反问开场，把整场分享的主线立住了：推理集群的优化不是一次性的配置调参，而是一条从部署形态、调度架构到资源切分的演进链路。</p>
<p>他把这条链路抽象成三个递进的阶段：</p>
<ul>
<li class=""><strong>Phase 0</strong>: vLLM as a process，单机进程，快速验证</li>
<li class=""><strong>Phase 1</strong>: vLLM as a workload，Kubernetes 上的标准化工作负载</li>
<li class=""><strong>Phase 2</strong>: vLLM + PD 分离，Prefill 与 Decode 解耦</li>
<li class=""><strong>Phase 3</strong>: vLLM + PD 分离 + GPU 虚拟化，把 Decode 阶段的空闲算力彻底榨干</li>
</ul>
<p>下面逐个阶段展开。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="phase-0vllm-as-a-process单机快速验证">Phase 0：vLLM as a process，单机快速验证<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#phase-0vllm-as-a-process%E5%8D%95%E6%9C%BA%E5%BF%AB%E9%80%9F%E9%AA%8C%E8%AF%81" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="Phase 0: vLLM as a process" src="https://project-hami.io/zh/assets/images/phase0-overview-9dd00d8757658ec1440ae5d37803854a.webp" width="2000" height="1125" class="img_ev3q"></p>
<p>第一阶段面向最朴素的场景：你手头有一台带 GPU 的服务器，想用最快的速度把 vLLM 推理服务跑起来做验证。</p>
<p>这个阶段没有编排、没有调度，vLLM 就是一个跑在单机上的进程。李孟轩给出了两种最常见的拉起方式。</p>
<p><img decoding="async" loading="lazy" alt="Phase 0: Conda 与 Docker 两种部署方式" src="https://project-hami.io/zh/assets/images/phase0-code-546d1117c16a3f40e9cfa1ab43508b42.webp" width="2000" height="1125" class="img_ev3q"></p>
<p><strong>Conda 方式</strong> 用虚拟环境直接启动，可以通过 <code>CUDA_VISIBLE_DEVICES</code> 指定 GPU 卡，配合 <code>--tensor-parallel-size</code> 开启张量并行：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">conda create </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-n</span><span class="token plain"> vllm </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">python</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token number" style="color:rgb(247, 140, 108)">3.11</span><span class="token plain"> </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-y</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">conda activate vllm</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># CUDA 12.1</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">pip </span><span class="token function" style="color:rgb(130, 170, 255)">install</span><span class="token plain"> torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">CUDA_VISIBLE_DEVICES</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token number" style="color:rgb(247, 140, 108)">0,1</span><span class="token plain">,2,3 python </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">-m</span><span class="token plain"> vllm.entrypoints.openai.api_server </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--model</span><span class="token plain"> meta-llama/Meta-Llama-3-70B-Instruct </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--host</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">0.0</span><span class="token plain">.0.0 </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--port</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">8000</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  --tensor-parallel-size </span><span class="token number" style="color:rgb(247, 140, 108)">4</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  --gpu-memory-utilization </span><span class="token number" style="color:rgb(247, 140, 108)">0.9</span><br></div></code></pre></div></div>
<p><strong>Docker 方式</strong> 则用官方镜像一行命令拉起，同样支持指定设备和显存利用率。</p>
<blockquote>
<p>这个阶段本质上是一个独立进程，只适合用在开发 / 测试场景做快速验证，并不是标准化的集群工作负载形态。</p>
</blockquote>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="phase-1vllm-as-a-workloadkubernetes-上的标准化部署">Phase 1：vLLM as a workload，Kubernetes 上的标准化部署<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#phase-1vllm-as-a-workloadkubernetes-%E4%B8%8A%E7%9A%84%E6%A0%87%E5%87%86%E5%8C%96%E9%83%A8%E7%BD%B2" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="Phase 1: vLLM as a workload" src="https://project-hami.io/zh/assets/images/phase1-overview-6eda820b46509142cc824777ca10c66e.webp" width="2000" height="1125" class="img_ev3q"></p>
<p>当场景从单机扩展到一个 GPU 集群，手动逐节点指定启动指令和分配 GPU 就不再现实了。第二阶段引入了 <strong>Kubernetes 作为统一编排底座</strong>: 把 vLLM 封装成一个标准的 Pod，写成 Deployment YAML，剩下的调度交给 K8s。</p>
<p><img decoding="async" loading="lazy" alt="Phase 1: 把 vLLM 写成 K8s Deployment" src="https://project-hami.io/zh/assets/images/phase1-k8s-27ee51ff8a6451c0cd27d1c0de622155.webp" width="2000" height="1125" class="img_ev3q"></p>
<p>核心就是一份 Deployment，在 <code>resources.limits</code> 里声明 <code>nvidia.com/gpu: 1</code>，K8s 会自动把负载调度到有空闲 GPU 的节点上：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">apiVersion</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> apps/v1</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">kind</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Deployment</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">metadata</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">server</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">namespace</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">inference</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">labels</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">app</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">spec</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># ...</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">containers</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">image</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> vllm/vllm</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">openai</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">latest</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">args</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">model</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> meta</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">llama/Meta</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">Llama</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">3</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">8B</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">Instruct</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">host</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"0.0.0.0"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">port</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"8000"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">utilization</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"0.9"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">max</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">model</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">len</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"8192"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">resources</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">        </span><span class="token key atrule">limits</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">nvidia.com/gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"32Gi"</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">          </span><span class="token key atrule">cpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"8"</span><br></div></code></pre></div></div>
<p>到这里，vLLM 已经是生产可用的标准化工作负载。但李孟轩强调：直接提交这类工作负载的方案对生产环境的适配性还不够，还有进一步优化的空间，于是引出第三阶段。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="phase-2vllm--pd-分离把-prefill-和-decode-解耦">Phase 2：vLLM + PD 分离，把 Prefill 和 Decode 解耦<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#phase-2vllm--pd-%E5%88%86%E7%A6%BB%E6%8A%8A-prefill-%E5%92%8C-decode-%E8%A7%A3%E8%80%A6" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="Phase 2: PD 分离" src="https://project-hami.io/zh/assets/images/phase2-pd-d2a2daefc6e0b59771af8b57479ced3c.webp" width="2000" height="1125" class="img_ev3q"></p>
<p><strong>PD 分离（Prefill / Decode Disaggregation）</strong> 是大模型推理走向生产级的关键优化。它的核心思路是：把推理的两个阶段拆到不同的负载上分别处理。</p>
<ul>
<li class=""><strong>Prefill 阶段</strong> 负责处理 prompt、生成首 token 和上下文对应的 KV Cache，是计算密集型</li>
<li class=""><strong>Decode 阶段</strong> 负责逐个吐出后续 token，是访存（带宽）密集型</li>
</ul>
<p>把两者拆开的最大价值在于：<strong>避免新请求的 Prefill 干扰正在进行的 Decode，从而稳定保障用户体感上的 TPOT（每个输出 token 的延迟）</strong>。</p>
<p>李孟轩介绍了落地 PD 分离的三种路径，从最轻量到最云原生。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="路径一vllm-原生-p2pncclconnector">路径一：vLLM 原生 P2pNcclConnector<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#%E8%B7%AF%E5%BE%84%E4%B8%80vllm-%E5%8E%9F%E7%94%9F-p2pncclconnector" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p><img decoding="async" loading="lazy" alt="Phase 2: P2pNcclConnector" src="https://project-hami.io/zh/assets/images/phase2-nccl-e63af57321da9bbd8eb7bbbab1a3a3ad.webp" width="2000" height="1125" class="img_ev3q"></p>
<p>最轻量的方式不需要任何额外组件，直接用 vLLM 原生能力。Prefill 实例作为 <code>kv_producer</code>、Decode 实例作为 <code>kv_consumer</code>，通过 <strong>NCCL P2P 连接器</strong> 在两者之间传输 KV Cache:</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># Prefill</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">CUDA_VISIBLE_DEVICES</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token number" style="color:rgb(247, 140, 108)">0</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">vllm serve meta-llama/Meta-Llama-3-8B-Instruct </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--port</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">8100</span><span class="token plain"> --enforce-eager </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  --kv-transfer-config </span><span class="token string" style="color:rgb(195, 232, 141)">'{</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">    "kv_connector": "P2pNcclConnector",</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">    "kv_role": "kv_producer",</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">    "kv_rank": 0,</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">    "kv_parallel_size": 2</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">  }'</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># Decode</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">CUDA_VISIBLE_DEVICES</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token number" style="color:rgb(247, 140, 108)">0</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">MASTER_ADDR</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token number" style="color:rgb(247, 140, 108)">192.168</span><span class="token plain">.1.10 </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">MASTER_PORT</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token number" style="color:rgb(247, 140, 108)">29500</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">vllm serve meta-llama/Meta-Llama-3-8B-Instruct </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--port</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">8200</span><span class="token plain"> --enforce-eager </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  --kv-transfer-config </span><span class="token string" style="color:rgb(195, 232, 141)">'{</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">    "kv_connector": "P2pNcclConnector",</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">    "kv_role": "kv_consumer",</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">    "kv_rank": 1,</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">    "kv_parallel_size": 2</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token string" style="color:rgb(195, 232, 141)">  }'</span><br></div></code></pre></div></div>
<p>这种方式上手最快，但当 Decode 副本变多时，就需要配套的请求路由能力。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="路径二mooncake脱离-k8s-也能用">路径二：Mooncake，脱离 K8s 也能用<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#%E8%B7%AF%E5%BE%84%E4%BA%8Cmooncake%E8%84%B1%E7%A6%BB-k8s-%E4%B9%9F%E8%83%BD%E7%94%A8" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p><img decoding="async" loading="lazy" alt="Phase 2: Mooncake" src="https://project-hami.io/zh/assets/images/phase2-mooncake-fc005e580f01efe66de6f357775b5b84.webp" width="2000" height="1125" class="img_ev3q"></p>
<p>如果不在 Kubernetes 环境里，<strong>Mooncake</strong> 提供了另一套方案。它包含三个核心模块：</p>
<ul>
<li class=""><strong>KV Cache / 请求路由</strong>: 决定请求该发到哪个 Decode 节点</li>
<li class=""><strong>Transfer Engine（对标 NCCL Connector）</strong>: 负责 KV Cache 的跨节点 P2P 传输</li>
<li class=""><strong>共享存储</strong>: 支撑 KV Cache 的共享</li>
</ul>
<p><img decoding="async" loading="lazy" alt="Phase 2: Mooncake 部署结构" src="https://project-hami.io/zh/assets/images/phase2-mooncake-code-60cc9b06298cc384ab425f06f8324461.webp" width="2000" height="1125" class="img_ev3q"></p>
<p>Mooncake 通过一个 Conductor 做集中调度，Prefill 和 Decode 各自起一个 transfer_engine，配合 <code>MooncakeConnector</code> 完成 KV Cache 的跨节点拷贝。它最大的特点是 <strong>可以脱离 K8s 独立运行</strong>，适配裸机部署的 PD 分离场景。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="路径三llm-d云原生生产级方案">路径三：LLM-D，云原生生产级方案<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#%E8%B7%AF%E5%BE%84%E4%B8%89llm-d%E4%BA%91%E5%8E%9F%E7%94%9F%E7%94%9F%E4%BA%A7%E7%BA%A7%E6%96%B9%E6%A1%88" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p><img decoding="async" loading="lazy" alt="Phase 2: LLM-D" src="https://project-hami.io/zh/assets/images/phase2-llmd-eacebc91a65c7c1e64ab3171e17a19cd.webp" width="2000" height="1125" class="img_ev3q"></p>
<p><strong>LLM-D</strong> 是面向 Kubernetes 生产环境的开源项目，可以理解为"云原生形态的 Mooncake 能力组件"。它的设计更贴合 K8s 生态：</p>
<ul>
<li class="">通过 <strong>CRD 定义推理池</strong> 里的 Prefill、Decode 节点属性</li>
<li class="">Router 基于 <strong>KV Cache 命中</strong> 和 <strong>prompt 语义相似度</strong> 两类策略分配请求，保障 TTFT</li>
<li class="">底层可兼容 Mooncake 连接器</li>
</ul>
<p><img decoding="async" loading="lazy" alt="Phase 2: LLM-D 架构细节" src="https://project-hami.io/zh/assets/images/phase2-llmd-detail-b328ab2ab5fc8dec55979b09d3506263.webp" width="2000" height="1125" class="img_ev3q"></p>
<p>李孟轩表示，LLM-D 相关的部署机制已经过诸多验证，是当前在 K8s 上落地 PD 分离比较成熟的开源选择。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="phase-3pd-分离--gpu-虚拟化把-decode-的空闲算力榨干">Phase 3：PD 分离 + GPU 虚拟化，把 Decode 的空闲算力榨干<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#phase-3pd-%E5%88%86%E7%A6%BB--gpu-%E8%99%9A%E6%8B%9F%E5%8C%96%E6%8A%8A-decode-%E7%9A%84%E7%A9%BA%E9%97%B2%E7%AE%97%E5%8A%9B%E6%A6%A8%E5%B9%B2" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="Phase 3: vLLM + PD 分离 + GPU 虚拟化" src="https://project-hami.io/zh/assets/images/phase3-overview-756bb213a665c29589c25e575897a9ef.webp" width="2000" height="1125" class="img_ev3q"></p>
<p>走到第三阶段，问题回到开场的那句反问：算力真的用满了吗？</p>
<p>关键洞察在于 Decode 阶段的瓶颈特征：<strong>当前主流硬件下，Decode 是访存（带宽）瓶颈，而不是算力瓶颈</strong>。也就是说，即使把业务并发拉满，GPU 的计算单元仍有大量空闲，算力资源并没有被充分利用。</p>
<p>既然 Decode 单卡吃不满，那就可以把多个 Decode 负载塞到同一张 GPU 上跑。这正是 <strong>HAMi GPU 虚拟化</strong> 发挥作用的场景。</p>
<p><img decoding="async" loading="lazy" alt="Phase 3: PD 分离 + GPU 虚拟化整体架构" src="https://project-hami.io/zh/assets/images/phase3-architecture-6a6ded4c2bf4baa38fd8b2386dc18a54.webp" width="2000" height="1125" class="img_ev3q"></p>
<p>整体架构是 <strong>vLLM + LLM-D 实现 PD 分离，再叠加 HAMi GPU 虚拟化</strong>:</p>
<ul>
<li class=""><strong>Prefill 节点</strong> 是计算密集型，保持独占 GPU 模式，在资源配置里直接指定 GPU 数量即可</li>
<li class=""><strong>Decode 节点</strong> 通过 HAMi 做 GPU 切片调度，只需在 Deployment 里指定 GPU 显存占比，就能让多个 Decode 实例共享同一张卡</li>
<li class="">上层由 <strong>llm-d router</strong> 负责请求路由，通过 HAMi Plugin 把请求分发到切分出的 GPU Slice 上</li>
</ul>
<p><img decoding="async" loading="lazy" alt="Phase 3: 优化效果" src="https://project-hami.io/zh/assets/images/phase3-result-80fe3eb2f1d1077494c27e72bd268fce.webp" width="2000" height="1125" class="img_ev3q"></p>
<p>通过 GPU 虚拟化把多个 Decode 负载调度到同一张 GPU，可以把 <strong>单卡承载的 Decode 实例能力提升约一倍</strong>。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="现场答疑精华">现场答疑精华<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#%E7%8E%B0%E5%9C%BA%E7%AD%94%E7%96%91%E7%B2%BE%E5%8D%8E" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>分享结束后的答疑环节同样信息量很大，几个核心结论值得单独拎出来。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="gpu-虚拟化复用-decode单卡降一点集群涨两成">GPU 虚拟化复用 Decode，单卡降一点，集群涨两成<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#gpu-%E8%99%9A%E6%8B%9F%E5%8C%96%E5%A4%8D%E7%94%A8-decode%E5%8D%95%E5%8D%A1%E9%99%8D%E4%B8%80%E7%82%B9%E9%9B%86%E7%BE%A4%E6%B6%A8%E4%B8%A4%E6%88%90" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>有参会者问：把多个 Decode 塞进一张卡，性能会不会受影响？</p>
<p>李孟轩的回答很坦诚：这其实是一个 <strong>妥协的做法</strong>，单卡内多个 Decode 负载会存在带宽竞争，单卡自身的解码吞吐会出现小幅下降。但从集群视角看，释放出来的空闲 GPU 可以承载更多业务负载，<strong>整体集群的综合吞吐能实现约 20% 的提升</strong>。在 GPU 越来越贵的当下，这笔账是划算的。</p>
<blockquote>
<p>「这个其实大部分情况下是一个妥协的做法，因为大家都知道现在的卡越来越贵嘛。」</p>
</blockquote>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="hami-软件虚拟化-vs-英伟达-mig">HAMi 软件虚拟化 vs 英伟达 MIG<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#hami-%E8%BD%AF%E4%BB%B6%E8%99%9A%E6%8B%9F%E5%8C%96-vs-%E8%8B%B1%E4%BC%9F%E8%BE%BE-mig" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>讨论中还对比了 HAMi 的 GPU 虚拟化与英伟达原生的 <strong>MIG（Multi-Instance GPU）</strong> 技术：</p>
<table><thead><tr><th>维度</th><th>HAMi GPU 虚拟化</th><th>英伟达 MIG</th></tr></thead><tbody><tr><td>限制层级</td><td>软件层调度</td><td>硬件层硬配额绑定</td></tr><tr><td>显存 / 算力 / 带宽</td><td>动态抢占，空闲资源可复用</td><td>切分时同步锁定上限，其他实例空闲也无法占用</td></tr><tr><td>调度灵活性</td><td>高，带宽抢占灵活</td><td>低，易出现性能浪费</td></tr></tbody></table>
<p>MIG 在切分实例时会同步限制各实例的显存、算力与带宽上限，调度灵活性差、容易浪费；而 HAMi 在软件层做限制，<strong>带宽抢占更灵活，可以充分利用显卡带宽</strong>。</p>
<blockquote>
<p>「用 HAMi 的话，它在软件层做的限制，所以带宽抢占是比较灵活的。」</p>
</blockquote>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="decode-阶段为何是访存瓶颈">Decode 阶段为何是访存瓶颈<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#decode-%E9%98%B6%E6%AE%B5%E4%B8%BA%E4%BD%95%E6%98%AF%E8%AE%BF%E5%AD%98%E7%93%B6%E9%A2%88" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>最后一个问题聚焦在瓶颈判断：即便用了 PD 分离，场景是不是大多仍属于访存瓶颈，还是部分环境能达到算力瓶颈？</p>
<p>结论很明确：<strong>当前多数生产环境下，Decode 阶段普遍是访存瓶颈，很难真正触达算力瓶颈</strong>。除非卡的访存带宽特别特别高，否则一旦带宽打满，算力想提也提不上去。这也正是 Phase 3"用 GPU 虚拟化复用 Decode 空闲算力"这一思路成立的根本前提。</p>
<blockquote>
<p>「你很难在解码阶段真正把算力达到瓶颈，因为带宽一旦打满了，算力你想提也提不上去。」</p>
</blockquote>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="一句话总结">一句话总结<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#%E4%B8%80%E5%8F%A5%E8%AF%9D%E6%80%BB%E7%BB%93" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>从 Phase 0 的单机进程，到 Phase 1 的 K8s 工作负载，到 Phase 2 的 PD 分离，再到 Phase 3 叠加 HAMi GPU 虚拟化，李孟轩给出的不是某个银弹参数，而是一套 <strong>可落地、可递进</strong> 的推理集群优化方法论：先把部署标准化，再把 Prefill / Decode 解耦，最后基于 Decode 是访存瓶颈这一事实，用 GPU 虚拟化把空闲算力彻底释放出来。</p>
<p><strong>算力效率从来不是单点能力，而是部署形态、调度架构与资源切分共同作用的结果。</strong> 这，大概就是"你的算力用的好么？"这个问题最好的回答。</p>
<p><img decoding="async" loading="lazy" alt="vLLM Meetup 现场合影" src="https://project-hami.io/zh/assets/images/vllm-meetup-group-photo-bf4720f9219d568744cc84659faabfba.webp" width="2400" height="1600" class="img_ev3q"></p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="资源链接">资源链接<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#%E8%B5%84%E6%BA%90%E9%93%BE%E6%8E%A5" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class=""><strong>PPT 下载</strong>: <a href="https://github.com/Project-HAMi/community/tree/main/talks/02-vllm-meetup-20260716" target="_blank" rel="noopener noreferrer" class="">分享 PPT</a></li>
<li class=""><strong>HAMi 项目</strong>: <a href="https://github.com/Project-HAMi/HAMi" target="_blank" rel="noopener noreferrer" class="">https://github.com/Project-HAMi/HAMi</a></li>
<li class=""><strong>密瓜智能 Dynamia</strong>: <a href="https://dynamia.ai/" target="_blank" rel="noopener noreferrer" class="">https://dynamia.ai</a></li>
<li class=""><strong>vLLM</strong>: <a href="https://github.com/vllm-project/vllm" target="_blank" rel="noopener noreferrer" class="">https://github.com/vllm-project/vllm</a></li>
<li class=""><strong>LLM-D</strong>: <a href="https://github.com/llm-d/llm-d" target="_blank" rel="noopener noreferrer" class="">https://github.com/llm-d/llm-d</a></li>
<li class=""><strong>Mooncake</strong>: <a href="https://github.com/kvcache-ai/Mooncake" target="_blank" rel="noopener noreferrer" class="">https://github.com/kvcache-ai/Mooncake</a></li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="关于分享者">关于分享者<a href="https://project-hami.io/zh/blog/vllm-meetup-shanghai-2026-recap#%E5%85%B3%E4%BA%8E%E5%88%86%E4%BA%AB%E8%80%85" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><strong>李孟轩</strong>，密瓜智能（Dynamia）联合创始人兼 CTO，HAMi 项目作者。长期致力于异构算力调度与 GPU 虚拟化技术，HAMi 已成为云原生 GPU 共享领域最具影响力的开源项目之一，并于近期晋级 CNCF 孵化项目。</p>]]></content>
        <author>
            <name>李孟轩</name>
            <uri>https://github.com/archlitchi</uri>
        </author>
        <category label="vLLM" term="vLLM"/>
        <category label="GPU 虚拟化" term="GPU 虚拟化"/>
        <category label="PD Disaggregation" term="PD Disaggregation"/>
        <category label="LLM-D" term="LLM-D"/>
        <category label="Mooncake" term="Mooncake"/>
        <category label="HAMi" term="HAMi"/>
        <category label="Inference Optimization" term="Inference Optimization"/>
        <category label="云原生" term="云原生"/>
        <category label="AI 基础设施" term="AI 基础设施"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[HAMi 晋升 CNCF 孵化项目]]></title>
        <id>https://project-hami.io/zh/blog/hami-cncf-incubating</id>
        <link href="https://project-hami.io/zh/blog/hami-cncf-incubating"/>
        <updated>2026-07-02T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[2026 年 7 月 2 日，CNCF 技术监督委员会（TOC）投票通过，HAMi 正式晋升为 CNCF Incubating（孵化）项目。]]></summary>
        <content type="html"><![CDATA[<p>我们很高兴地宣布：<strong>2026 年 7 月 2 日，HAMi 正式晋升为 CNCF Incubating（孵化）项目</strong>，CNCF 技术监督委员会以全票赞成通过了<a href="https://github.com/cncf/toc/issues/1775" target="_blank" rel="noopener noreferrer" class="">本次孵化投票</a>。</p>
<p>这是 HAMi 继 2024 年 8 月作为 Sandbox 项目加入 CNCF 后的又一个重要里程碑，意味着 CNCF 技术监督委员会（TOC）认可 HAMi 已具备成熟的技术与安全实践、活跃的社区、真实的生产采用与开放的生态集成。</p>
<p>衷心感谢主导本次尽职调查的 CNCF TOC 主席 <a href="https://www.linkedin.com/feed/update/urn:li:activity:7478594903292399616/" target="_blank" rel="noopener noreferrer" class="">Karena Angell</a> 与 TOC 成员王泽峰（Kevin Wang），为项目完成技术评审的 TAG-Runtime，以及所有提供反馈的采用者。</p>
<p>HAMi 是面向 Kubernetes 的异构算力虚拟化与调度中间件，不改驱动、不改应用，通过 CUDA API 劫持（HAMi-core）在软件层实现 GPU 虚拟化，精细切分 GPU 显存与算力，让多个工作负载安全共享同一张加速器。项目由 <a href="https://github.com/wawa0210" target="_blank" rel="noopener noreferrer" class="">张潇</a>、<a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">李孟轩</a> 等维护者发起，在 CNCF 治理下由全球开发者共建。</p>
<p>今天，HAMi 已被数百家组织采用，覆盖 NVIDIA、昇腾、寒武纪、海光、摩尔线程、燧原、昆仑芯、MetaX、AWS Neuron、Vastai 等十余种加速器（见<a class="" href="https://project-hami.io/zh/docs/userguide/device-supported">支持设备列表</a>），并以 <a class="" href="https://project-hami.io/zh/case-studies">CNCF 案例研究</a>的形式记录了招商银行、SNOW Corp.、蔚来、贝壳、DaoCloud、顺丰科技、PREP EDU 等的落地实践，横跨金融、汽车、出行、物流、教育与云服务。HAMi-core 已与 Kubernetes 默认调度器、<a href="https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_gpu_sharing.md" target="_blank" rel="noopener noreferrer" class="">Volcano</a>、<a href="https://github.com/kubernetes-sigs/kueue" target="_blank" rel="noopener noreferrer" class="">Kueue</a>、<a href="https://koordinator.sh/docs/user-manuals/device-scheduling-gpu-share-with-hami" target="_blank" rel="noopener noreferrer" class="">Koordinator</a> 以及 <a class="" href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler">KAI Scheduler</a> 完成集成。</p>
<p>这一里程碑离不开全球贡献者与社区的共建：HAMi 至今已汇聚来自几十个国家的近 500 名贡献者。<a href="https://www.dynamia.ai/" target="_blank" rel="noopener noreferrer" class="">密瓜智能（Dynamia）</a>、<a href="https://www.daocloud.io/" target="_blank" rel="noopener noreferrer" class="">DaoCloud</a>、<a href="https://www.4paradigm.com/" target="_blank" rel="noopener noreferrer" class="">第四范式</a>、<a href="https://www.nvidia.com/" target="_blank" rel="noopener noreferrer" class="">NVIDIA</a>、<a href="https://www.huawei.com/" target="_blank" rel="noopener noreferrer" class="">华为云</a>，以及众多个人开发者、用户企业与生态伙伴，都作出了不可替代的贡献。</p>
<p>进入孵化阶段是更高的起点。社区将继续投入上游研发、生态适配与生产级落地，与全球伙伴一起，让 HAMi 成为 AI 时代异构算力管理的开放基础设施。</p>
<p>欢迎在 <a href="https://github.com/Project-HAMi/HAMi" target="_blank" rel="noopener noreferrer" class="">GitHub</a> 与<a class="" href="https://project-hami.io/zh/community">社区</a>加入我们。</p>]]></content>
        <author>
            <name>HAMi 社区</name>
        </author>
        <category label="CNCF" term="CNCF"/>
        <category label="Incubating" term="Incubating"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[HAMi 亮相 KubeCon + CloudNativeCon India 2026：将 GPU 共享带给社区]]></title>
        <id>https://project-hami.io/zh/blog/kubecon-india-2026-recap</id>
        <link href="https://project-hami.io/zh/blog/kubecon-india-2026-recap"/>
        <updated>2026-06-29T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[回顾 HAMi 在孟买 KubeCon + CloudNativeCon India 2026 的亮相，包括 Keynote 现场演示、展台 Demo，以及围绕 Kubernetes 原生 GPU 共享的社区交流。]]></summary>
        <content type="html"><![CDATA[<p>2026 年 6 月 18-19 日，<a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-india/" target="_blank" rel="noopener noreferrer" class="">KubeCon + CloudNativeCon India 2026</a> 在印度孟买举行，来自云原生生态的实践者、平台工程师、AI 基础设施团队和开源贡献者齐聚一堂。随着 AI 成为本届大会最重要的主题之一，HAMi 展示了 Kubernetes 原生 GPU 共享如何帮助组织提升加速器利用率，同时保持工作负载隔离和运维灵活性。</p>
<p>从开场 Keynote 到展台现场演示，再到与工程团队的技术讨论，本次活动凸显了一个越来越明确的行业关注点：让昂贵的 GPU 基础设施真正适用于多租户 AI 工作负载。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="ai-基础设施成为大会焦点">AI 基础设施成为大会焦点<a href="https://project-hami.io/zh/blog/kubecon-india-2026-recap#ai-%E5%9F%BA%E7%A1%80%E8%AE%BE%E6%96%BD%E6%88%90%E4%B8%BA%E5%A4%A7%E4%BC%9A%E7%84%A6%E7%82%B9" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>本届大会一开始就高度聚焦 AI 基础设施。在开场 Keynote 中，Chris Aniszczyk 强调，云原生技术正在成为下一代 AI 平台的基础。</p>
<p>HAMi 在 KubeCon India 的亮相正是对这一趋势的直接回应：通过现场演示，展示 Kubernetes 如何在多个团队和推理工作负载之间共享 GPU 资源，而不必为每个工作负载分配一整张 GPU 或一套独立集群。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="keynote-demokubernetes-上的多租户-ai-推理">Keynote Demo：Kubernetes 上的多租户 AI 推理<a href="https://project-hami.io/zh/blog/kubecon-india-2026-recap#keynote-demokubernetes-%E4%B8%8A%E7%9A%84%E5%A4%9A%E7%A7%9F%E6%88%B7-ai-%E6%8E%A8%E7%90%86" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="Saiyam Pathak 在 KubeCon India Keynote 中展示 HAMi GPU 共享演示" src="https://project-hami.io/zh/assets/images/keynote-demo-f326b6bc6215fa08b82d89cb09d81ce8.jpg" width="1440" height="1080" class="img_ev3q"></p>
<p>在 <a href="https://kccncind2026.sched.com/event/2NgRQ/keynote-from-platforms-to-ai-factories-has-kubernetes-solved-it-saiyam-pathak-principal-developer-advocate-vcluster" target="_blank" rel="noopener noreferrer" class="">Saiyam Pathak 的 Keynote</a> 中，HAMi 支撑了一场展示 GPU 共享能力的现场演示。</p>
<p>演示架构包括：</p>
<ul>
<li class="">运行 Kubernetes 控制平面的 MacBook</li>
<li class="">作为 GPU 工作节点的 NVIDIA DGX Spark</li>
<li class="">用于租户隔离的 vCluster</li>
<li class="">用于细粒度 GPU 分配的 HAMi</li>
<li class="">多个 AI 推理工作负载同时运行在共享 GPU 上</li>
</ul>
<p><img decoding="async" loading="lazy" alt="演示架构：两个基于 vCluster 的团队通过 HAMi 共享一张 DGX Spark GPU" src="https://project-hami.io/zh/assets/images/demo-architecture-zh-8b1e48c7b3686e23e7819e315cbe266a.png" width="1672" height="941" class="img_ev3q"></p>
<p>为了让概念更直观，演示使用了带有孟买本地特色的应用场景，包括本地火车、dabbawala 送餐员和 kaali peeli 出租车。移动应用采集的图片会被路由到 Kubernetes，并交由运行在共享 GPU 基础设施上的 AI 模型完成推理。</p>
<p>这场 Keynote 并不只是解释 GPU 虚拟化概念，而是展示了多个独立 AI 团队如何共享同一张物理加速卡，同时保持彼此隔离的 Kubernetes 环境。</p>
<p>该演示说明，HAMi 与 vCluster 结合后，可以支撑资源高效、云原生的多租户 AI 平台。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="展台现场演示">展台现场演示<a href="https://project-hami.io/zh/blog/kubecon-india-2026-recap#%E5%B1%95%E5%8F%B0%E7%8E%B0%E5%9C%BA%E6%BC%94%E7%A4%BA" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>在 Project Pavilion，参观者体验了两场动手演示，展示 Kubernetes 上真实的 AI 推理场景。</p>
<p>如果你希望复现其中一部分 vLLM 工作流，可以参考 <a class="" href="https://project-hami.io/zh/tutorials/labs/hami-vllm">HAMi vLLM 实验</a>。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="demo-1基于-vllm-与-sglang-的生产级-ai-推理">Demo 1：基于 vLLM 与 SGLang 的生产级 AI 推理<a href="https://project-hami.io/zh/blog/kubecon-india-2026-recap#demo-1%E5%9F%BA%E4%BA%8E-vllm-%E4%B8%8E-sglang-%E7%9A%84%E7%94%9F%E4%BA%A7%E7%BA%A7-ai-%E6%8E%A8%E7%90%86" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>第一场演示展示了在 Kubernetes 上使用 vLLM 和 SGLang 进行生产级 LLM serving，并由 HAMi 管理异构硬件上的 GPU 分配。演示使用了两种模型交付方式：直接从 Hugging Face 加载模型，以及通过 Kubernetes initContainers 分发 KitOps ModelKits 打包的模型，从而展示 AI 平台中灵活的部署策略。</p>
<p>推理栈运行在：</p>
<ul>
<li class="">Nebius 提供的 NVIDIA H100 GPU</li>
<li class="">Reza Jelveh 提供的 NVIDIA A10 GPU</li>
</ul>
<p>参观者可以看到：</p>
<ul>
<li class="">生产级 LLM serving</li>
<li class="">Kubernetes 原生部署工作流</li>
<li class="">感知资源的 GPU 调度</li>
<li class="">跨推理工作负载的高效 GPU 共享</li>
<li class="">通过现代推理引擎完成端到端请求路由</li>
</ul>
<p>通过将 HAMi 与 vLLM、SGLang 等流行推理框架结合，这场演示展示了组织如何在提升 GPU 利用率的同时部署可扩展的 AI 服务。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="demo-2动态-gpu-共享">Demo 2：动态 GPU 共享<a href="https://project-hami.io/zh/blog/kubecon-india-2026-recap#demo-2%E5%8A%A8%E6%80%81-gpu-%E5%85%B1%E4%BA%AB" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>第二场演示聚焦 HAMi 的 GPU 虚拟化能力。</p>
<p>参观者了解了多个 AI 工作负载如何通过显式资源控制共存于同一张物理 GPU 上，相关能力包括：</p>
<ul>
<li class="">分数 GPU 分配</li>
<li class="">动态资源分配</li>
<li class="">灵活调度策略</li>
<li class="">GPU 虚拟化</li>
<li class="">Kubernetes 原生部署</li>
</ul>
<p>参观者还通过 NVIDIA 工具验证资源分配情况，并实时观察推理请求如何流经共享基础设施。</p>
<p>这两场演示共同展示了 Kubernetes 如何从简单的 GPU 分配，进一步演进到面向现代 AI 平台的智能加速器调度。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="展台交流">展台交流<a href="https://project-hami.io/zh/blog/kubecon-india-2026-recap#%E5%B1%95%E5%8F%B0%E4%BA%A4%E6%B5%81" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>除演示之外，HAMi 展台也成为技术交流的中心，吸引了工程师、创业者、平台团队、CTO 以及正在探索 AI 基础设施的学生。</p>
<p>虽然参会者来自不同领域，但许多讨论都围绕同一个问题展开：</p>
<blockquote>
<p>组织如何在更清晰的隔离和更可预测的工作负载放置基础上，共享昂贵的 GPU 资源？</p>
</blockquote>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="hyperverge面向多租户-ai-的显存隔离">HyperVerge：面向多租户 AI 的显存隔离<a href="https://project-hami.io/zh/blog/kubecon-india-2026-recap#hyperverge%E9%9D%A2%E5%90%91%E5%A4%9A%E7%A7%9F%E6%88%B7-ai-%E7%9A%84%E6%98%BE%E5%AD%98%E9%9A%94%E7%A6%BB" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>其中一次深入交流来自 HyperVerge。</p>
<p>他们的工程团队分享了此前使用 GPU 共享方案的经历：一些方案缺少推理工作负载之间清晰的 GPU 显存隔离控制。</p>
<p>对于在共享基础设施上服务多个客户的组织来说，这会带来显著的运维挑战。</p>
<p>这次讨论聚焦 HAMi 如何处理：</p>
<ul>
<li class="">GPU 显存隔离</li>
<li class="">智能调度</li>
<li class="">Bin-packing 策略</li>
<li class="">Spread 调度</li>
<li class="">多租户工作负载放置</li>
</ul>
<p>对于企业级 AI 平台而言，将高效 GPU 利用率与更强的工作负载隔离控制结合起来，具有很高价值。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="企业级调度讨论">企业级调度讨论<a href="https://project-hami.io/zh/blog/kubecon-india-2026-recap#%E4%BC%81%E4%B8%9A%E7%BA%A7%E8%B0%83%E5%BA%A6%E8%AE%A8%E8%AE%BA" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>来自 BlackRock 的工程师关注 HAMi 的调度算法，以及它如何在异构 GPU 工作负载之间做出放置决策。</p>
<p>类似地，InfraCloud 团队也讨论了 HAMi 如何集成到现有 Kubernetes 环境中，并对比了传统 Kubernetes Device Plugin、动态资源分配（DRA）和 HAMi 调度模型。</p>
<p>这些讨论反映了企业级 Kubernetes 平台中的一个更广泛趋势：</p>
<blockquote>
<p>组织正在积极寻找更智能的方式，以消除 GPU 利用率不足问题，同时支持日益多样化的 AI 工作负载。</p>
</blockquote>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="展台背后的社区">展台背后的社区<a href="https://project-hami.io/zh/blog/kubecon-india-2026-recap#%E5%B1%95%E5%8F%B0%E8%83%8C%E5%90%8E%E7%9A%84%E7%A4%BE%E5%8C%BA" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><img decoding="async" loading="lazy" alt="HAMi 社区贡献者在 KubeCon India Project Pavilion 展台" src="https://project-hami.io/zh/assets/images/hami-booth-9ca2b7a7ded4a6ba2af5f69c7465e0d8.jpg" width="1280" height="960" class="img_ev3q"></p>
<p>开源项目的成功离不开社区。</p>
<p>感谢所有帮助 HAMi 展台顺利运行的人：</p>
<ul>
<li class=""><a href="https://github.com/maishivamhoo123" target="_blank" rel="noopener noreferrer" class="">Shivam Kumar</a></li>
<li class=""><a href="https://github.com/satyampsoni" target="_blank" rel="noopener noreferrer" class="">Satyam Soni</a></li>
<li class=""><a href="https://github.com/rudrakshkarpe" target="_blank" rel="noopener noreferrer" class="">Rudraksh Karpe</a></li>
<li class=""><a href="https://github.com/shivaylamba" target="_blank" rel="noopener noreferrer" class="">Shivay Lamba</a></li>
<li class=""><a href="https://github.com/fishman" target="_blank" rel="noopener noreferrer" class="">Reza Jelveh</a></li>
</ul>
<p>特别感谢 <a href="https://github.com/rootsongjc" target="_blank" rel="noopener noreferrer" class="">Jimmy Song</a> 准备展台材料，并在整个活动期间支持社区。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="展望">展望<a href="https://project-hami.io/zh/blog/kubecon-india-2026-recap#%E5%B1%95%E6%9C%9B" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>KubeCon India 期间的交流强化了一个清晰信号。</p>
<p>组织正在从传统的“每个团队一张 GPU”模式中走出来，积极寻找 Kubernetes 原生方案，以实现高效、可扩展、具备实际隔离控制的 GPU 共享。</p>
<p>随着 AI 基础设施持续演进，HAMi 这类技术将在帮助组织实现以下目标时发挥越来越重要的作用：</p>
<ul>
<li class="">提升 GPU 利用率</li>
<li class="">降低基础设施成本</li>
<li class="">支撑多租户 AI 平台</li>
<li class="">在 Kubernetes 上部署生产级推理</li>
<li class="">更高效地扩展加速器基础设施</li>
</ul>
<p>感谢每一位来到展台、参加 Keynote、提供反馈并贡献想法的朋友。</p>
<p>我们很高兴继续构建 Kubernetes GPU 基础设施的未来，也期待在下一届 KubeCon 再见。</p>]]></content>
        <author>
            <name>HAMi 社区</name>
        </author>
        <category label="KubeCon" term="KubeCon"/>
        <category label="GPU" term="GPU"/>
        <category label="Kubernetes" term="Kubernetes"/>
        <category label="AI" term="AI"/>
        <category label="India" term="India"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[HAMi-core 被 NVIDIA KAI Scheduler 采用：GPU 共享正式迈入硬隔离时代]]></title>
        <id>https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler</id>
        <link href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler"/>
        <updated>2026-06-20T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[两项核心 PR 已合并进入 NVIDIA KAI Scheduler 主干，HAMi 的 GPU 显存硬隔离能力已成为 KAI Scheduler v0.16.4 的内置特性。云原生 GPU 资源调度正式从「软共享」迈入「硬隔离」时代。]]></summary>
        <content type="html"><![CDATA[<blockquote>
<p>本文中的集成对象严格来说是 HAMi-core，而非完整 HAMi 平台。KAI Scheduler 保留自身调度能力，引入 HAMi-core 提供 GPU Memory Isolation 能力。</p>
</blockquote>
<p>2026 年 6 月，两项核心 PR 正式合并进入 NVIDIA KAI Scheduler 主干。HAMi 的 GPU 显存硬隔离能力已作为内置特性随 KAI Scheduler v0.16.4 发布，云原生 GPU 资源调度正式从「软共享」迈入「硬隔离」时代。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="什么是-kai-scheduler">什么是 KAI Scheduler？<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#%E4%BB%80%E4%B9%88%E6%98%AF-kai-scheduler" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p><a href="https://github.com/kai-scheduler/KAI-Scheduler" target="_blank" rel="noopener noreferrer" class="">KAI Scheduler</a> 是 NVIDIA 开源的 Kubernetes 原生 AI 工作负载调度器。它的前身是 Run<!-- -->:ai<!-- --> 的调度引擎，NVIDIA 于 2024 年底收购 Run<!-- -->:ai<!-- --> 后，在 2025 年 4 月以 Apache 2.0 协议开源，现已成为 CNCF Sandbox 项目。</p>
<p>Kubernetes 默认调度器为无状态服务设计，把 GPU 当 CPU 核一样调度，每个 Pod 独占整张 GPU，没有 gang scheduling，没有团队公平性，没有拓扑感知。KAI Scheduler 正是为解决这些 AI 场景特有的调度问题而生：</p>
<ul>
<li class=""><strong>PodGroup（Gang Scheduling）</strong>：分布式训练的多个 Pod 必须同时启动，否则全部等待。避免 7 张 GPU 被占着却跑不起来的尴尬。</li>
<li class=""><strong>Queue（层级公平调度）</strong>：按部门/团队分配 GPU 配额，支持借用和回收，实现多团队共享集群的公平调度。</li>
<li class=""><strong>Fractional GPU（GPU 分片共享）</strong>：多个工作负载共享同一张 GPU，按比例或按显存大小分配。</li>
<li class=""><strong>Topology-Aware Placement</strong>：感知 GPU 间互联拓扑，把紧耦合的训练任务放在同一节点或同一 NVLink 域内。</li>
<li class=""><strong>Elastic Workloads</strong>：任务可以在最小和最大 Pod 数之间弹性伸缩，随集群负载动态调整。</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="kai-scheduler-的最后一公里gpu-硬隔离">KAI Scheduler 的「最后一公里」：GPU 硬隔离<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#kai-scheduler-%E7%9A%84%E6%9C%80%E5%90%8E%E4%B8%80%E5%85%AC%E9%87%8Cgpu-%E7%A1%AC%E9%9A%94%E7%A6%BB" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>KAI Scheduler 的 GPU 分片共享（Fractional GPU）功能虽然很强大，但是有一个关键限制：</p>
<p>KAI Scheduler 的 GPU 共享是「协作式」的：调度器确保请求的显存份额加起来不超过 GPU 总量，但<strong>不物理阻止</strong>某个工作负载超额使用显存。假如一个容器请求 2000 MiB 显存，但是在该容器中仍然可以通过 <code>nvidia-smi</code> 和 CUDA API 看到并使用完整 GPU 显存。</p>
<p>这对于开发测试环境通常可以接受。但在生产环境的多租户场景下，这就成了致命短板：</p>
<ul>
<li class="">无法防止工作负载超额使用显存，导致 OOM 或互相干扰</li>
<li class="">多租户之间缺乏真正的资源隔离保障</li>
<li class="">无法精确控制每个容器的 GPU 显存上限</li>
</ul>
<p>这正是 HAMi 的核心能力所在。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="什么是-hami">什么是 HAMi？<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#%E4%BB%80%E4%B9%88%E6%98%AF-hami" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>HAMi 是 CNCF 沙箱项目，专注于异构 AI 算力虚拟化中间件。其核心能力是通过 CUDA 拦截库（HAMi-core）在 CUDA API 层强制限制 GPU 显存和算力，使工作负载无法超额使用所分配的份额。</p>
<p>简单理解 HAMi 的定位：</p>
<ul>
<li class=""><strong>KAI Scheduler</strong> = 决定「谁在什么时候用什么 GPU」（调度层）</li>
<li class=""><strong>HAMi</strong> = 确保「用了就只有这么多，不能多占」（隔离层）</li>
</ul>
<p>两者结合，才能实现真正意义上的生产级 GPU 共享。HAMi 支持 NVIDIA GPU、华为昇腾 NPU、寒武纪 MLU、海光 DCU、昆仑芯 XPU 等多种异构加速卡，是云原生 GPU 虚拟化领域覆盖最广的开源方案。详细的异构加速卡支持信息请参考 <a class="" href="https://project-hami.io/zh/docs/userguide/device-supported">HAMi 官方文档</a>。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="集成架构hami--kai-scheduler-如何协作">集成架构：HAMi + KAI Scheduler 如何协作<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#%E9%9B%86%E6%88%90%E6%9E%B6%E6%9E%84hami--kai-scheduler-%E5%A6%82%E4%BD%95%E5%8D%8F%E4%BD%9C" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>整个集成方案采用松耦合设计，KAI Scheduler 和 HAMi 各司其职、独立部署：</p>
<figure class="mermaid-figure"><figcaption>HAMi + KAI Scheduler 集成流程</figcaption></figure>
<p>工作流程分为四个阶段：</p>
<ol>
<li class=""><strong>调度</strong>：KAI Scheduler 将 Pod 调度到合适的 GPU 节点</li>
<li class=""><strong>环境变量注入</strong>：KAI 的 Admission 组件根据 <code>gpu-fraction</code> 或 <code>gpu-memory</code> 注解，向容器注入 <code>CUDA_DEVICE_MEMORY_LIMIT</code> 环境变量</li>
<li class=""><strong>库文件注入</strong>：kai-resource-isolator 的 MutatingWebhook 自动注入 HAMi-core 库的 volume mount 和 <code>ld.so.preload</code> 配置</li>
<li class=""><strong>运行时隔离</strong>：容器启动后，<code>libvgpu.so</code> 拦截所有 CUDA 内存分配调用，根据环境变量强制执行显存上限</li>
</ol>
<figure class="mermaid-figure"><figcaption>GPU 显存隔离：HAMi 接入前后对比</figcaption></figure>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="部署方式">部署方式<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#%E9%83%A8%E7%BD%B2%E6%96%B9%E5%BC%8F" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>在 KAI Scheduler 中集成 HAMi 也极其简单，仅两步。</p>
<p><strong>第一步：安装 KAI Scheduler</strong>，启用 GPU 共享并激活 <code>hamicore</code> 插件：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm </span><span class="token function" style="color:rgb(130, 170, 255)">install</span><span class="token plain"> kai-scheduler oci://ghcr.io/nvidia/kai-scheduler </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--version</span><span class="token plain"> v0.16.4 </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--set</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">global.gpuSharing</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">true </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--set</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">binder.plugins.hamicore.enabled</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">true </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--namespace</span><span class="token plain"> kai-scheduler --create-namespace</span><br></div></code></pre></div></div>
<p><strong>第二步：部署 kai-resource-isolator</strong>。它把 HAMi-core 库以 DaemonSet 形式分发到每个 GPU 节点，并通过 MutatingWebhook 向共享 GPU 的 Pod 注入卷挂载：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm </span><span class="token function" style="color:rgb(130, 170, 255)">install</span><span class="token plain"> kai-resource-isolator oci://docker.io/projecthami/kai-resource-isolator </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--namespace</span><span class="token plain"> kai-resource-isolator --create-namespace </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--version</span><span class="token plain"> </span><span class="token number" style="color:rgb(247, 140, 108)">1.0</span><span class="token plain">.0-chart</span><br></div></code></pre></div></div>
<p>Chart 版本号带 <code>-chart</code> 后缀（如 <code>1.0.0-chart</code>），可用版本见 <a href="https://hub.docker.com/r/projecthami/kai-resource-isolator/tags" target="_blank" rel="noopener noreferrer" class="">Docker Hub</a>；更多定制项参考 <a href="https://github.com/Project-HAMi/KAI-resource-isolator" target="_blank" rel="noopener noreferrer" class="">kai-resource-isolator 仓库</a>。</p>
<p>部署完成后，任何使用 <code>gpu-fraction</code> 或 <code>gpu-memory</code> 注解的 Pod 都会自动获得显存隔离。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="使用方式">使用方式<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#%E4%BD%BF%E7%94%A8%E6%96%B9%E5%BC%8F" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>以请求 4096 MiB 显存为例，给 Pod 打上 <code>gpu-memory</code> 注解并指定 <code>kai-scheduler</code> 调度器即可：</p>
<div class="language-yaml codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-yaml codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token key atrule">apiVersion</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> v1</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">kind</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> Pod</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">metadata</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">sharing</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">with</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">isolation</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">labels</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">kai.scheduler/queue</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> default</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">queue</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">annotations</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token key atrule">gpu-memory</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"4096"</span><span class="token plain"> </span><span class="token comment" style="color:rgb(105, 112, 152);font-style:italic"># 单位 MiB，不带后缀</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain"></span><span class="token key atrule">spec</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">schedulerName</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> kai</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">scheduler</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token key atrule">containers</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">    </span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain"> </span><span class="token key atrule">name</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> gpu</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">workload</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">image</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> nvidia/cuda</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain">12.9.2</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">base</span><span class="token punctuation" style="color:rgb(199, 146, 234)">-</span><span class="token plain">ubuntu24.04</span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">      </span><span class="token key atrule">command</span><span class="token punctuation" style="color:rgb(199, 146, 234)">:</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(199, 146, 234)">[</span><span class="token string" style="color:rgb(195, 232, 141)">"sleep"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">,</span><span class="token plain"> </span><span class="token string" style="color:rgb(195, 232, 141)">"infinity"</span><span class="token punctuation" style="color:rgb(199, 146, 234)">]</span><br></div></code></pre></div></div>
<p>Pod 启动后，容器内的 <code>nvidia-smi</code> 只会显示分配到的显存，而非整张 GPU 的完整显存，资源隔离得到验证。</p>
<h4 class="anchor anchorTargetStickyNavbar_Vzrq" id="按需关闭隔离opt-out">按需关闭隔离（Opt-out）<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#%E6%8C%89%E9%9C%80%E5%85%B3%E9%97%AD%E9%9A%94%E7%A6%BBopt-out" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h4>
<ul>
<li class=""><strong>单个 Pod</strong>：添加注解 <code>kai-resource-isolator.io/inject: "false"</code></li>
<li class=""><strong>整个命名空间</strong>：添加标签 <code>kai-resource-isolator.io/webhook=ignore</code></li>
</ul>
<h4 class="anchor anchorTargetStickyNavbar_Vzrq" id="显存数值精度">显存数值精度<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#%E6%98%BE%E5%AD%98%E6%95%B0%E5%80%BC%E7%B2%BE%E5%BA%A6" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h4>
<p><code>gpu-memory</code> 注解接受<strong>整数 MiB</strong>（无单位后缀）。KAI Scheduler 内部会把它换算成两位小数的 GPU 分数，再乘以 GPU 总显存得出实际强制上限。因此 <code>nvidia-smi</code> 看到的值可能与请求值略有出入，例如在 15360 MiB 的 T4 上请求 <code>4096</code>，会四舍五入为 <code>0.27</code> 分数，最终强制上限为 <code>4147m</code>。</p>
<div class="theme-admonition theme-admonition-tip admonition_xJq3 alert alert--success"><div class="admonitionHeading_Gvgb"><span class="admonitionIcon_Rf37"><svg viewBox="0 0 12 16"><path fill-rule="evenodd" d="M6.5 0C3.48 0 1 2.19 1 5c0 .92.55 2.25 1 3 1.34 2.25 1.78 2.78 2 4v1h5v-1c.22-1.22.66-1.75 2-4 .45-.75 1-2.08 1-3 0-2.81-2.48-5-5.5-5zm3.64 7.48c-.25.44-.47.8-.67 1.11-.86 1.41-1.25 2.06-1.45 3.23-.02.05-.02.11-.02.17H5c0-.06 0-.13-.02-.17-.2-1.17-.59-1.83-1.45-3.23-.2-.31-.42-.67-.67-1.11C2.44 6.78 2 5.65 2 5c0-2.2 2.02-4 4.5-4 1.22 0 2.36.42 3.22 1.19C10.55 2.94 11 3.94 11 5c0 .66-.44 1.78-.86 2.48zM4 14h5c-.23 1.14-1.3 2-2.5 2s-2.27-.86-2.5-2z"></path></svg></span>提示</div><div class="admonitionContent_BuS1"><p>本文只讲重点。完整步骤（前置条件、安装、调度隔离 Pod、用 <code>nvidia-smi</code> 验证隔离、按需关闭）见 <a class="" href="https://project-hami.io/zh/docs/next/userguide/kai-scheduler/how-to-use-kai-scheduler">如何在 KAI Scheduler 中使用 HAMi</a> 文档。</p></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="开源协作从提案到合并">开源协作：从提案到合并<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#%E5%BC%80%E6%BA%90%E5%8D%8F%E4%BD%9C%E4%BB%8E%E6%8F%90%E6%A1%88%E5%88%B0%E5%90%88%E5%B9%B6" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>此次集成是开源社区协作的典范，历时超过一年，涉及 HAMi 团队与 NVIDIA KAI Scheduler 团队的紧密配合：</p>
<table><thead><tr><th>时间</th><th>里程碑</th><th>参与者</th></tr></thead><tbody><tr><td>2025 年 4 月</td><td><a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">@archlitchi</a> 提交 PR #60「Resource isolation design」提出资源隔离设计；NVIDIA 团队评审方案、讨论架构设计并确定分工；社区讨论敲定技术方案：KAI 负责环境变量注入，HAMi 负责资源隔离组件</td><td><a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">@archlitchi</a>（HAMi）、<a href="https://github.com/romanbaron" target="_blank" rel="noopener noreferrer" class="">@romanbaron</a>、<a href="https://github.com/enoodle" target="_blank" rel="noopener noreferrer" class="">@enoodle</a>（NVIDIA）及双方团队与社区</td></tr><tr><td>2026 年 4 月</td><td><a href="https://github.com/FouoF" target="_blank" rel="noopener noreferrer" class="">@FouoF</a> 提交 PR #1504，实现 GPU_MEMORY_LIMIT binder 插件</td><td><a href="https://github.com/FouoF" target="_blank" rel="noopener noreferrer" class="">@FouoF</a>（HAMi）</td></tr><tr><td>2026 年 5 月 28 日</td><td>PR #1504 合并进入 KAI Scheduler 主干</td><td><a href="https://github.com/davidLif" target="_blank" rel="noopener noreferrer" class="">@davidLif</a>（NVIDIA）merge</td></tr><tr><td>2026 年 6 月</td><td><a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">@archlitchi</a> 完成用户文档、e2e 测试，PR #60 通过全部 review</td><td><a href="https://github.com/archlitchi" target="_blank" rel="noopener noreferrer" class="">@archlitchi</a>（HAMi）</td></tr><tr><td>2026 年 6 月 9 日</td><td>PR #60 正式合并进入 KAI Scheduler 主干</td><td><a href="https://github.com/davidLif" target="_blank" rel="noopener noreferrer" class="">@davidLif</a>, <a href="https://github.com/gshaibi" target="_blank" rel="noopener noreferrer" class="">@gshaibi</a>（NVIDIA）approve</td></tr></tbody></table>
<p>此次集成的社区反响十分热烈：</p>
<ul>
<li class="">PR #60 多位社区开发者表达了对此功能的强烈需求</li>
<li class="">Thanh Tung Dao 等用户积极跟进进展，期待 v0.16.0 版本发布</li>
<li class="">社区讨论覆盖了从技术方案到部署模型的各个层面</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="对-hami-生态的意义">对 HAMi 生态的意义<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#%E5%AF%B9-hami-%E7%94%9F%E6%80%81%E7%9A%84%E6%84%8F%E4%B9%89" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="验证了-hami-技术路线的正确性">验证了 HAMi 技术路线的正确性<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#%E9%AA%8C%E8%AF%81%E4%BA%86-hami-%E6%8A%80%E6%9C%AF%E8%B7%AF%E7%BA%BF%E7%9A%84%E6%AD%A3%E7%A1%AE%E6%80%A7" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>HAMi 的核心能力（CUDA 拦截和 GPU 显存硬隔离）被 NVIDIA 官方调度器采纳，这是对 HAMi 技术成熟度的有力背书。NVIDIA 团队选择将 HAMi-core 作为 KAI Scheduler GPU 共享的资源隔离方案，而非自行开发，说明 HAMi 在这个领域的方案已经是最优解。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="拓展了生态版图">拓展了生态版图<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#%E6%8B%93%E5%B1%95%E4%BA%86%E7%94%9F%E6%80%81%E7%89%88%E5%9B%BE" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>在此之前，HAMi 已与多个 Kubernetes 调度器完成集成。此次整合将覆盖范围拓展到 NVIDIA 官方调度器：</p>
<figure class="mermaid-figure"><figcaption>HAMi-core 调度器生态</figcaption></figure>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="为用户创造实际价值">为用户创造实际价值<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#%E4%B8%BA%E7%94%A8%E6%88%B7%E5%88%9B%E9%80%A0%E5%AE%9E%E9%99%85%E4%BB%B7%E5%80%BC" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>对于同时使用 KAI Scheduler 和 HAMi 的用户，这次集成解决了最迫切的需求。正如社区用户 Thanh Tung Dao 所说：</p>
<blockquote>
<p>"We're currently using KAI Scheduler to handle our ML workloads, but we have a new requirement: we need to enforce strict vGPU restrictions (memory/compute isolation) at the pod level. I know HAMi excels at this."</p>
</blockquote>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="现已可用kai-scheduler-v0164-中的-hami-资源隔离">现已可用：KAI Scheduler v0.16.4 中的 HAMi 资源隔离<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#%E7%8E%B0%E5%B7%B2%E5%8F%AF%E7%94%A8kai-scheduler-v0164-%E4%B8%AD%E7%9A%84-hami-%E8%B5%84%E6%BA%90%E9%9A%94%E7%A6%BB" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<p>两项核心 PR 已全部合并进入 KAI Scheduler 主干，并随 v0.16.4 正式发布。用户只需在 Helm 安装 KAI Scheduler 时启用 GPU 共享与 <code>hamicore</code> 插件，即可获得 HAMi 资源隔离：</p>
<div class="language-bash codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#bfc7d5;--prism-background-color:#292d3e"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-bash codeBlock_bY9V thin-scrollbar" style="color:#bfc7d5;background-color:#292d3e"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#bfc7d5"><span class="token plain">helm </span><span class="token function" style="color:rgb(130, 170, 255)">install</span><span class="token plain"> kai-scheduler oci://ghcr.io/nvidia/kai-scheduler </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--version</span><span class="token plain"> v0.16.4 </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--set</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">global.gpuSharing</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">true </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--set</span><span class="token plain"> </span><span class="token assign-left variable" style="color:rgb(191, 199, 213)">binder.plugins.hamicore.enabled</span><span class="token operator" style="color:rgb(137, 221, 255)">=</span><span class="token plain">true </span><span class="token punctuation" style="color:rgb(199, 146, 234)">\</span><span class="token plain"></span><br></div><div class="token-line" style="color:#bfc7d5"><span class="token plain">  </span><span class="token parameter variable" style="color:rgb(191, 199, 213)">--namespace</span><span class="token plain"> kai-scheduler --create-namespace</span><br></div></code></pre></div></div>
<p><code>global.gpuSharing=true</code> 开启 GPU 共享，<code>binder.plugins.hamicore.enabled=true</code> 激活 <code>hamicore</code> 插件，由其为共享 GPU 的容器注入 <code>CUDA_DEVICE_MEMORY_LIMIT</code> 环境变量；再配合节点侧的 kai-resource-isolator 强制执行，即可实现显存硬隔离（完整步骤见上文「部署方式」）。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="后续规划">后续规划<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#%E5%90%8E%E7%BB%AD%E8%A7%84%E5%88%92" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<ul>
<li class="">配合 KAI Scheduler 新版本发布，完善用户文档和使用指南</li>
<li class="">探索 GPU 计算单元（SM）隔离的支持</li>
<li class="">持续优化 HAMi-core 在大规模集群场景下的性能表现</li>
</ul>
<hr>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="相关链接">相关链接<a href="https://project-hami.io/zh/blog/hami-core-adopted-by-nvidia-kai-scheduler#%E7%9B%B8%E5%85%B3%E9%93%BE%E6%8E%A5" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class="">HAMi 项目：<a href="https://github.com/Project-HAMi/HAMi" target="_blank" rel="noopener noreferrer" class="">github.com/Project-HAMi/HAMi</a></li>
<li class="">KAI Scheduler：<a href="https://github.com/kai-scheduler/KAI-Scheduler" target="_blank" rel="noopener noreferrer" class="">github.com/kai-scheduler/KAI-Scheduler</a></li>
<li class="">PR #60（Resource isolation design）：<a href="https://github.com/kai-scheduler/KAI-Scheduler/pull/60" target="_blank" rel="noopener noreferrer" class="">github.com/kai-scheduler/KAI-Scheduler/pull/60</a></li>
<li class="">PR #1504（GPU_MEMORY_LIMIT binder）：<a href="https://github.com/kai-scheduler/KAI-Scheduler/pull/1504" target="_blank" rel="noopener noreferrer" class="">github.com/kai-scheduler/KAI-Scheduler/pull/1504</a></li>
<li class="">HAMi 资源隔离用户文档：<a href="https://github.com/kai-scheduler/KAI-Scheduler/blob/main/docs/gpu-sharing/hami/README.md" target="_blank" rel="noopener noreferrer" class="">docs/gpu-sharing/hami/README.md</a></li>
<li class="">kai-resource-isolator：<a href="https://github.com/Project-HAMi/KAI-resource-isolator" target="_blank" rel="noopener noreferrer" class="">github.com/Project-HAMi/KAI-resource-isolator</a></li>
</ul>]]></content>
        <author>
            <name>HAMi 社区</name>
        </author>
        <category label="HAMi" term="HAMi"/>
        <category label="NVIDIA" term="NVIDIA"/>
        <category label="KAI Scheduler" term="KAI Scheduler"/>
        <category label="GPU 共享" term="GPU 共享"/>
        <category label="硬隔离" term="硬隔离"/>
        <category label="Kubernetes" term="Kubernetes"/>
        <category label="云原生" term="云原生"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[HAMi 即将亮相 KubeCon + CloudNativeCon India 2026]]></title>
        <id>https://project-hami.io/zh/blog/hami-at-kubecon-india-2026</id>
        <link href="https://project-hami.io/zh/blog/hami-at-kubecon-india-2026"/>
        <updated>2026-06-10T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[HAMi 将在 KubeCon India 2026 设立 Project Pavilion 展台，Keynote 环节将有使用 NVIDIA DGX Spark 的 GPU 共享现场演示。]]></summary>
        <content type="html"><![CDATA[<p>HAMi 即将亮相 <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-india/" target="_blank" rel="noopener noreferrer" class="">KubeCon + CloudNativeCon India 2026</a>，6 月 18–19 日，孟买。继 <a class="" href="https://project-hami.io/zh/blog/hami-at-kubecon-eu-2026">KubeCon Europe 2026</a> 之后，来自印度的 HAMi 社区贡献者和 CNCF Ambassador 将在孟买带来 GPU 共享演示和社区交流。</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="hami--kubecon-india-2026-活动概览">HAMi @ KubeCon India 2026 活动概览<a href="https://project-hami.io/zh/blog/hami-at-kubecon-india-2026#hami--kubecon-india-2026-%E6%B4%BB%E5%8A%A8%E6%A6%82%E8%A7%88" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="1-keynote-from-platforms-to-ai-factories">1. Keynote: From Platforms to AI Factories<a href="https://project-hami.io/zh/blog/hami-at-kubecon-india-2026#1-keynote-from-platforms-to-ai-factories" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<ul>
<li class=""><strong>演讲者</strong>：<a href="https://kccncind2026.sched.com/speaker/saiyam_pathak" target="_blank" rel="noopener noreferrer" class="">Saiyam Pathak</a></li>
<li class=""><strong>时间</strong>：6 月 18 日 11:03–11:13 IST · Jasmine 2（3 层）</li>
<li class=""><strong><a href="https://kccncind2026.sched.com/event/2NgRQ" target="_blank" rel="noopener noreferrer" class="">议程链接</a></strong></li>
</ul>
<p>本次 Keynote 将探讨 Kubernetes 中的 GPU 共享问题。<strong>HAMi 将一张物理 GPU 虚拟化为多个切片</strong>，每个切片拥有独立的显存预算。现场演示将使用 MacBook 连接 NVIDIA DGX Spark，单张 Blackwell GPU 同时运行两个开源大模型。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="2-hami-展台">2. HAMi 展台<a href="https://project-hami.io/zh/blog/hami-at-kubecon-india-2026#2-hami-%E5%B1%95%E5%8F%B0" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<ul>
<li class=""><strong>展台 T-10</strong> · Project Pavilion · Jasmine Hall 1（3 层）</li>
<li class=""><strong>时间</strong>：6 月 18 日 下午 3:30–7:30 IST</li>
</ul>
<p>欢迎前来体验 GPU 共享演示，了解 GPU 分配、DRA 集成，领取 HAMi 贴纸。</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="3-展台团队">3. 展台团队<a href="https://project-hami.io/zh/blog/hami-at-kubecon-india-2026#3-%E5%B1%95%E5%8F%B0%E5%9B%A2%E9%98%9F" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h3>
<p>展台由 HAMi 社区贡献者和 CNCF Ambassador 负责：</p>
<ul>
<li class=""><a href="https://www.linkedin.com/in/satyampsoni/" target="_blank" rel="noopener noreferrer" class=""><strong>Satyam Soni</strong></a>，CNCF Ambassador，KCD Delhi 组织者，Kubernetes Release Engineering</li>
<li class=""><a href="https://www.linkedin.com/in/rudrakshkarpe/" target="_blank" rel="noopener noreferrer" class=""><strong>Rudraksh Karpe</strong></a>，Simplismart Forward Deployed Engineer，GSoC Mentor</li>
<li class=""><a href="https://www.linkedin.com/in/shivaylamba/" target="_blank" rel="noopener noreferrer" class=""><strong>Shivay Lamba</strong></a>，GSoC Org Admin（Jenkins），Full Stack &amp; AI</li>
<li class=""><a href="https://www.linkedin.com/in/shivam-kumar-559417290/" target="_blank" rel="noopener noreferrer" class=""><strong>Shivam Kumar</strong></a>，HAMi (CNCF) 贡献者</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="联系我们">联系我们<a href="https://project-hami.io/zh/blog/hami-at-kubecon-india-2026#%E8%81%94%E7%B3%BB%E6%88%91%E4%BB%AC" class="hash-link" aria-label="跳转到头部" title="跳转到头部" translate="no">​</a></h2>
<ul>
<li class="">Star <a href="https://github.com/Project-HAMi/HAMi" target="_blank" rel="noopener noreferrer" class="">GitHub</a></li>
<li class="">加入 <a href="https://discord.gg/Amhy7XmbNq" target="_blank" rel="noopener noreferrer" class="">Discord</a> 或 <a href="https://cloud-native.slack.com/archives/C07T10BU4R2/" target="_blank" rel="noopener noreferrer" class="">CNCF Slack (#hami-dev)</a></li>
</ul>
<p>孟买见！🇮🇳 HAMi 展台：T-10</p>]]></content>
        <author>
            <name>HAMi 社区</name>
        </author>
        <category label="KubeCon" term="KubeCon"/>
        <category label="GPU" term="GPU"/>
        <category label="Kubernetes" term="Kubernetes"/>
        <category label="AI" term="AI"/>
        <category label="India" term="India"/>
    </entry>
</feed>