• 反射：UE 通过 UClass/UProperty 等系统在运行时提供类型信息和动态访问能力。
  UE 的反射系统是通过 UHT 工具和特定宏实现的代码生成机制。你用 UCLASS 标记类、UPROPERTY 标记变量、UFUNCTION 标记函数，这些宏会被 UHT 识别。
  UHT 在编译前扫描这些标记，生成.generated.h 和.cpp 文件，里面包含类的反射注册代码，比如 StaticClass () 函数和 UClass 对象的构造逻辑。
  生成的代码会把类信息注册到引擎全局的 GObjectClasses 数组里，让引擎在运行时能动态获取类结构、调用函数或访问属性，这支撑了蓝图交互、垃圾回收等核心功能。

因为 UE 需要在运行时动态处理代码信息。比如蓝图可视化编程，引擎得通过反射知道 C++ 类有哪些函数和变量，才能让蓝图调用它们。

比如你在 C++ 里写了一个角色类，里面有个 UFUNCTION 标记的跳跃函数 Jump ()。没有反射的话，蓝图编辑器根本不知道这个 Jump () 函数存在，因为编译后的机器码里，函数名和参数这些信息都被优化掉了。
有了反射，UHT 会在编译时为这个 Jump () 函数生成反射元数据，包括函数名、参数类型、返回值，以及它属于哪个类。引擎运行时能通过这些元数据，在蓝图编辑器里把 Jump () 函数显示出来，你才能拖拽节点调用它。
如果后续你在 C++ 里给 Jump () 加了一个高度参数，反射系统会自动更新元数据，蓝图里对应的函数节点也会同步显示出新参数，整个过程不需要手动写任何蓝图和 C++ 交互的绑定代码。

回退操作 Command 模式（轻量级）：**

1
2
3
4

每次操作封装为 ICommand { Do(); Undo(); }
维护 undoStack 和 redoStack
执行操作 → 压入 undoStack，清空 redoStack
Undo → 弹出 undoStack，执行 Undo()，压入 redoStack

2. Snapshot 模式（适用于复杂场景）：

1
2
3
4

操作前序列化整个对象状态的快照
Undo 时直接恢复快照
优点：实现简单，不容易出 bug
缺点：内存开销大

实际项目中的混合方案：

• 简单属性修改 → Command 模式（记录 oldValue/newValue）
• 复杂操作（节点图变更、场景编辑）→ Snapshot 或 Diff 模式
• 合并机制：连续同类操作合并（如拖拽 Slider 时合并为一条记录）

UE智能指针对比表

关键区别说明

1. 管理对象边界：前三种严格用于UObject派生类，依赖UE垃圾回收系统；后三种用于非UObject类型，靠手动内存管理机制。

2. UObject指针细分：

• TObjectPtr是强引用，会让UObject保持存活，是日常开发的首选。
• TWeakObjectPtr是弱引用，当UObject被标记为回收时，指针会自动置空，常用在UI控件引用角色对象这类场景。
• TSoftObjectPtr存储的是资源路径而非直接内存地址，对象未加载时可异步加载，适合开放世界游戏引用远处的资源。

3. 非UObject指针细分：

• TSharedPtr通过引用计数共享对象，当引用计数为0时自动释放内存，但需注意手动避免循环引用。
• TUniquePtr是独占式指针，不允许复制，只能通过移动语义转移所有权，性能开销最小。
• TWeakPtr需要绑定到TSharedPtr使用，当TSharedPtr释放对象后，TWeakPtr会自动失效，解决循环引用问题。

ECS 架构是什么？和传统 OOP 有什么区别？

核心概念：

• Entity：ID 标识，不存数据
• Component：纯数据（Position, Velocity, Health…）
• System：纯逻辑（MovementSystem 遍历所有 Position+Velocity 组件）

核心区别：OOP 以对象为核心，数据与逻辑封装在类中，易形成复杂继承树；ECS 将数据与逻辑分离，实体为组件容器，系统批量处理同类组件，数据连续存储提升缓存效率，支持动态组合与并行计算。
UE5 Mass 系统案例：作为 ECS 实现，Mass 用 “片段” 存储实体数据，“处理器” 统一处理逻辑。如《黑客帝国》Demo 中的万人级 crowd 模拟，通过将角色位置、速度等数据打包连续存储，移动处理器可批量更新所有角色坐标，性能远超传统 Actor 方案。

堆Stack 栈heap

• 堆（Heap）：动态分配内存，大小不固定，生命周期由程序员控制，访问速度较慢，适合存储大对象或需要在运行时确定大小的数据。（没有固定的存取顺序）
• 栈（Stack）：自动分配内存，大小固定，生命周期由函数调用控制，访问速度快，适合存储局部变量和函数参数。（有固定的存取顺序，后进先出）

Function Calling 的原理是什么？你在项目中怎么用的？

原理： LLM 不直接执行函数，而是 输出结构化的函数调用意图（函数名 + 参数），由宿主程序解析并执行。

RAG 是什么？你是怎么实现的？

RAG（Retrieval-Augmented Generation） = 先检索相关文档，再让 LLM 基于检索结果回答。

ControlNet 是什么？它解决了什么问题？

参考答案：

ControlNet 为预训练 Diffusion Model 添加 空间控制能力。

解决的问题： 原始 Text-to-Image 无法精确控制生成图像的构图、姿态、边缘等空间结构。

原理：

• 在 Stable Diffusion 的 U-Net 每个 Block 上添加一个并行的 “Zero Convolution” 分支
• 输入额外的条件图（边缘检测/Canny、深度图、姿态/OpenPose、法线贴图等）
• 训练时只训练 ControlNet 分支，冻结原始模型

常见 ControlNet 类型：

• Canny Edge：控制轮廓
• Depth：控制深度结构
• OpenPose：控制人物姿态
• Segment：控制区域分割
• Scribble：控制草图

LoRA 是什么？为什么它很受欢迎？

LoRA（Low-Rank Adaptation） 是一种参数高效微调方法。
LoRA 是一种参数高效的大模型微调技术，核心是冻结原模型权重，仅训练少量低秩矩阵来模拟任务适配所需的参数更新。它参数量仅为全量微调的 0.1%-1%，大幅降低显存占用和训练成本，且推理时可合并权重无额外延迟。
在游戏领域，能快速微调图生图模型生成风格统一的角色装备、场景素材，或微调对话模型让 NPC 生成符合设定的自然台词，适配小团队高效开发需求。

MVC、MVP、MVVM 的区别是什么？

• MVP的Preseter和MVVM的ViewModel在职责上非常相似，都是处理业务逻辑和数据交互的中介，但MVVM通过数据绑定机制让ViewModel直接更新View，减少了Presenter中大量的UI更新代码，使得代码更简洁、可测试性更强。MVVM适合复杂UI交互较多的项目，而MVP则更适合简单UI或需要严格分离测试的场景。

GPU 渲染流水线的完整阶段？

参考答案：

GPU 渲染管线（Rendering Pipeline）是 GPU 执行图形渲染的完整流程：

应用阶段（CPU 侧）：

1
2
3
4

1. 应用阶段（Application Stage）
   → 视锥体裁剪（Frustum Culling）
   → 批次合批（Draw Call Batching）
   → 输出 Draw Call + 顶点数据到 GPU

几何阶段（GPU 顶点着色器）：

1
2
3
4
5
6
7
8
9

2. 顶点着色器（Vertex Shader）
   → 模型空间 → 世界空间 → 视图空间 → 齐次裁剪空间
   → 顶点变换：LocalMatrix × WorldMatrix × ViewMatrix × ProjectionMatrix

3. 曲面细分（Tessellation，可选）
   → Hull Shader → Tessellator → Domain Shader

4. 几何着色器（Geometry Shader，可选）
   → 以图元为单位处理，可生成/销毁图元

光栅化阶段（Rasterization）：

1
2
3
4
5
6
7
8
9
10

5. 图元装配 & 裁剪
   → Clipping（齐次空间裁剪）
   → Perspective Divide → NDC → Viewport Transform

6. 背面剔除（Back-face Culling）
   → 根据顶 点环绕顺序（顺时针/逆时针）剔除背面

7. 光栅化（Rasterization）
   → 离散化：点/线/三角形 → 片段（Fragment）
   → 视口变换：NDC → Screen Space

片段/像素阶段：

1
2
3
4
5
6
7
8

8. 片段着色器（Fragment / Pixel Shader）
   → 逐像素着色：光照计算、纹理采样、颜色输出

9. 逐片段操作（Per-Fragment Operations）
   → 深度测试（Depth Test / Z-Test）
   → 模板测试（Stencil Test）
   → 混合（Alpha Blending）
   → 输出到 Framebuffer

UE “Reflection” Concept

• Reflection: UE provides runtime type information and dynamic access capabilities through systems like UClass and UProperty.
  UE’s reflection system is a code-generation mechanism implemented via the UHT tool and specific macros. You mark classes with UCLASS, variables with UPROPERTY, and functions with UFUNCTION — these macros are recognized by UHT.
  Before compilation, UHT scans these markers and generates .generated.h and .cpp files containing the reflection registration code for each class, such as the StaticClass() function and the construction logic for UClass objects.
  The generated code registers class information into the engine’s global GObjectClasses array, enabling the engine at runtime to dynamically retrieve class structure, invoke functions, or access properties — which in turn powers blueprint interaction, garbage collection, and other core features.

This exists because UE needs to dynamically process code information at runtime. Blueprint visual scripting, for example, requires the engine to know via reflection which functions and variables a C++ class exposes so that blueprints can call them.

Say you write a character class in C++ with a Jump() function marked with UFUNCTION. Without reflection, the blueprint editor has no idea Jump() exists — the function name, parameters, and all that metadata get optimized away in the compiled machine code.
With reflection, UHT generates reflection metadata for Jump() at compile time, including its name, parameter types, return value, and which class it belongs to. At runtime, the engine uses this metadata to surface Jump() in the blueprint editor as a callable node you can drag in.
If you later add a height parameter to Jump() in C++, the reflection system automatically updates the metadata, and the corresponding blueprint node syncs up to show the new parameter — no manual binding code between blueprint and C++ required.

Undo/Redo — Command Pattern (Lightweight)

1
2
3
4

Wrap each operation as ICommand { Do(); Undo(); }
Maintain undoStack and redoStack
Execute operation → push to undoStack, clear redoStack
Undo → pop from undoStack, call Undo(), push to redoStack

2. Snapshot Pattern (for complex scenarios):

1
2
3
4

Serialize a snapshot of the entire object state before each operation
On Undo, restore the snapshot directly
Pros: simple to implement, less prone to bugs
Cons: high memory overhead

Hybrid approach for real projects:

• Simple property edits → Command pattern (record oldValue/newValue)
• Complex operations (node graph changes, scene edits) → Snapshot or Diff pattern
• Merge mechanism: collapse consecutive operations of the same type (e.g., dragging a slider merges into a single history entry)

UE Smart Pointer Comparison

Key Distinctions

1. Object boundary: The first three are strictly for UObject-derived classes and rely on UE’s garbage collection system; the last three are for non-UObject types and use manual memory management.

2. UObject pointer breakdown:

   • TObjectPtr is a strong reference that keeps the UObject alive — the go-to choice for everyday development.
   • TWeakObjectPtr is a weak reference; when a UObject is marked for collection, the pointer is automatically nulled. Common in scenarios like UI widgets holding references to character objects.
   • TSoftObjectPtr stores a resource path rather than a direct memory address. The asset can be asynchronously loaded when it isn’t in memory, making it ideal for open-world games referencing distant assets.

3. Non-UObject pointer breakdown:

   • TSharedPtr shares an object via reference counting, automatically freeing memory when the count reaches zero. Be mindful of circular references — they must be avoided manually.
   • TUniquePtr is an exclusive pointer: no copying allowed, ownership transfers only through move semantics. Lowest performance overhead.
   • TWeakPtr must be bound to a TSharedPtr. Once TSharedPtr releases the object, TWeakPtr automatically becomes invalid, resolving circular reference issues.

What is ECS Architecture? How Does It Differ from Traditional OOP?

ECS essentially saves CPU time on data lookups.

Core concepts:

• Entity: an ID only, stores no data
• Component: pure data (Position, Velocity, Health…)
• System: pure logic (MovementSystem iterates all Position+Velocity components)

Core difference: OOP centers on objects — data and logic are encapsulated in classes, which tends to grow complex inheritance trees. ECS separates data from logic: entities are containers for components, systems process batches of the same component type, contiguous data storage improves cache efficiency, and the architecture naturally supports dynamic composition and parallel computation.
UE5 Mass system example: as an ECS implementation, Mass stores entity data in “fragments” and unifies logic in “processors.” The Matrix Awakens demo’s crowd simulation of thousands of characters packs position, velocity, and other data into contiguous storage, letting the movement processor batch-update all character coordinates — performance that far exceeds the traditional Actor approach.

Stack vs. Heap

• Heap: Dynamically allocated memory, variable size, lifetime controlled by the programmer, slower access. Suitable for large objects or data whose size is determined at runtime. (No fixed access order.)
• Stack: Automatically allocated memory, fixed size, lifetime controlled by the function call, fast access. Suitable for local variables and function parameters. (Fixed access order: last in, first out.)

What Is Function Calling and How Have You Used It in Projects?

How it works: The LLM doesn’t execute functions directly — it outputs a structured function-call intent (function name + arguments), which the host program parses and executes.

What Is RAG and How Did You Implement It?

RAG (Retrieval-Augmented Generation) = retrieve relevant documents first, then have the LLM answer based on those retrieved results.

What Is ControlNet and What Problem Does It Solve?

Reference answer:

ControlNet adds spatial control capabilities to a pretrained Diffusion Model.

The problem it solves: Raw Text-to-Image generation cannot precisely control the composition, pose, edges, or other spatial structure of generated images.

How it works:

• A parallel “Zero Convolution” branch is added to each block of Stable Diffusion’s U-Net
• Additional conditioning images are fed as input (edge detection / Canny, depth maps, pose / OpenPose, normal maps, etc.)
• During training, only the ControlNet branch is trained; the original model weights are frozen

Common ControlNet types:

• Canny Edge: controls outlines
• Depth: controls depth structure
• OpenPose: controls human pose
• Segment: controls region segmentation
• Scribble: controls sketch-based guidance

What Is LoRA and Why Is It So Popular?

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method.
LoRA freezes the original model weights and trains only a small number of low-rank matrices to approximate the parameter updates needed for task adaptation. The trainable parameter count is just 0.1%–1% of full fine-tuning, drastically reducing VRAM usage and training cost. During inference, the weights can be merged with the base model, adding no extra latency.
In game development, LoRA lets you quickly fine-tune an image-to-image model to generate stylistically consistent character equipment and environment assets, or fine-tune a dialogue model so NPCs produce setting-appropriate natural dialogue — a great fit for small teams that need to move fast.

What Is the Difference Between MVC, MVP, and MVVM?

MVP’s Presenter and MVVM’s ViewModel are very similar in responsibility — both act as intermediaries handling business logic and data interaction. The key difference is that MVVM’s data-binding mechanism lets ViewModel update the View directly, eliminating the large amount of UI-update code you’d write in a Presenter. This makes the code more concise and testable. MVVM suits projects with complex, frequent UI interactions; MVP fits simpler UIs or scenarios where strict test isolation is needed.

What Are the Complete Stages of the GPU Rendering Pipeline?

Reference answer:

The GPU Rendering Pipeline is the full process by which a GPU executes graphics rendering:

Application Stage (CPU side):

1
2
3
4

1. Application Stage
   → Frustum Culling
   → Draw Call Batching
   → Outputs Draw Calls + vertex data to GPU

Geometry Stage (GPU vertex shaders):

1
2
3
4
5
6
7
8
9

2. Vertex Shader
   → Model Space → World Space → View Space → Homogeneous Clip Space
   → Vertex transform: LocalMatrix × WorldMatrix × ViewMatrix × ProjectionMatrix

3. Tessellation (optional)
   → Hull Shader → Tessellator → Domain Shader

4. Geometry Shader (optional)
   → Processes per-primitive, can emit or discard primitives

Rasterization Stage:

1
2
3
4
5
6
7
8
9
10

5. Primitive Assembly & Clipping
   → Clipping (homogeneous space clipping)
   → Perspective Divide → NDC → Viewport Transform

6. Back-face Culling
   → Discards back-facing primitives based on vertex winding order (CW/CCW)

7. Rasterization
   → Discretization: points / lines / triangles → Fragments
   → Viewport Transform: NDC → Screen Space

Fragment / Pixel Stage:

1
2
3
4
5
6
7
8

8. Fragment Shader (Pixel Shader)
   → Per-pixel shading: lighting calculations, texture sampling, color output

9. Per-Fragment Operations
   → Depth Test (Z-Test)
   → Stencil Test
   → Alpha Blending
   → Output to Framebuffer

Series intro: “The Kung-Fu Manual for Landing AI in Game Production” records the road I took—every step of it paid for in blood and tears.
In Form 1, «Within Easy Reach», we taught AI to recognize every asset in a project. It knows which Jiangnan-style low shrub shrub_jn_01a refers to. But that is not enough: recognizing every piece on the board does not mean understanding the formation. In Form 2, «Read the Board, Discern the Formation», we teach AI to understand the “formation” an artist has arranged across a map.

Note: almost every article in this series is handcrafted. If you only care about the result and not the implementation, feel free to skip Chapters 3 through 6.

1. Origin: A Scene AI Cannot Understand

In Form 1, semantic encoding taught the AI to recognize individual assets. This time, the question is: where am I?

As everyone knows, an LLM does not understand a scene. Even if you hand it a screenshot, it still cannot read the elaborate formation an artist has built:

① "What kind of visual style does this area have?"     → AI: Make up something plausible.② "Find the treasure chest beside a staircase."     → AI: Traverse the World Outliner and inspect objects one by one. Which one is beside a staircase? No idea.③ "Here is a screenshot. Where is this location in my open world?"     → AI: What on earth am I looking at?④ "Find a cave I built out of piled rocks."     → AI: @#￥%……&*

Those are the problems this article sets out to solve. Map understanding turned out to contain too much material for one article, so I am splitting it into two: this one is about understanding the map, while the next will cover vector search across the entire map.

After this round of training, your AI will be able to understand a map in the real sense of the word, much as a person does, and memorize every scene and position it has sampled. For viewpoints that were never captured, I plan to infer them by interpolating between neighboring views, ultimately enabling millisecond-level lookup at any position in a large open world.

2. What Does It Mean to Understand a Scene?

Consider what happens when a person sees a scene.

First, they see the game world: “Huh, it is pitch-dark in here.” (visual)

Then they sense the spatial relationships: “Wait, there is some light ahead.” (spatial)

Finally, they understand what the scene means: “Oh—this must be the White Bone Demon’s cave.” (semantic)

How would they describe the scene to someone else?

I just found a place. You enter it by pressing E near Black Well. It is even darker than the cave at Flower-Fruit Mountain. I think it must be the hidden boss White Bone Demon’s lair.

From that description, I separate understanding from expression:

Difficulty: 🟢 easy ｜ 🟡 medium ｜ 🟠 hard ｜ 🔴 extremely hard. The harder cells are difficult for AI both to understand and to express accurately.

The vertical axis—the perception layer—corresponds to three abilities an AI needs in order to understand a scene:

• Visual information — What does this area look like? Style, density, texture, and aesthetic impression.
• Spatial information — How are A and B arranged in space? Who contains whom, what touches or intersects what, what supports what, and what occludes what: topology + direction + distance + support + occlusion.
  Topological relations—RCC-8’s DC / EC / PO / EQ / TPP / NTPP and so on—are already a mature area of qualitative spatial reasoning; see Cohn and Renz’s classic survey, Qualitative Spatial Representation and Reasoning. Support and occlusion are often treated separately in vision research, as in Biederman’s 1982 discussion of Support / Interposition. In an engine, however, they ultimately come from the same geometric data, so this framework groups them together by data source.
• Semantic information — What is A doing here, and what narrative role does it serve?
  1. Object meaning — What does this individual object imply?
  2. Compositional meaning — What does a group of objects mean together? This is mereology: for example, a gap hollowed out from a pile of rocks becomes a cave.
  3. Design semantics — This scene is run-down because it is meant to depict a village suffering through famine.
  4. Co-occurrence — Do A and B often appear together? Which pairings are stable?
     This level touches several different fields at once: distributional semantics, RDF for knowledge representation, and FrameNet for frame semantics. Co-occurrence is learned from frequency or vectors, not calculated from geometry. Modeling how the current state affects the next state would move further into Markov models. I treat co-occurrence as part of semantics because it expresses scene logic above the level of individual geometry. That topic is somewhat beyond the scope of this article.

The horizontal axis—the expression layer, meaning whether the system can restate what it has learned, covers three strategies:

• Reference corresponds to deixis: shared attention anchors both parties to the same object even without coordinates.
• Comparison uses vague quantifiers and vague language. It need not produce an exact number; it can instead communicate direction, degree, and an implied baseline.
• Metaphor corresponds to conceptual metaphor or cross-domain mapping: shared cultural experience is used to express space, texture, or function that would otherwise be difficult to state directly.

An ideal AI scene-understanding tool should, in theory, cover every cell in that 3 × 3 matrix.

A good agent also needs a long-term scene memory: if we discussed something before, it should be able to retrieve it when the subject comes up again.

Now let us talk about implementation.

3. Stop Talking Only About MCP: It Is a Bottomless Pit

What we actually need is a translation layer between engine data and the LLM: something that extracts whatever the LLM wants to know from the engine, then translates it into a form the model can use. MCP is the obvious first thought.

MCP has indeed been extremely popular for a while.

But MCP is not a pleasant job if what you want is a fast, dramatic payoff. It demands a great deal of engineering and tuning. Over the past few months I have spent an enormous amount of effort debugging it, even developing a tuning methodology and staring at telemetry every day. Honestly, the result is still not as good as I want.

Why?

If you have used any UE MCP tools, you have probably encountered several problems in the scene-understanding layer:

1. It cannot truly understand the style I am asking about.
   A person can look at a field and describe whether the grass is sparse or dense. The tool cannot “understand” or summarize that impression.

2. Context explodes as the tool count grows geometrically.
   Many MCP implementations expose interfaces originally designed for UI directly to the LLM. That is like handing a swordsman a kitchen knife: usable, but awkward. UI was designed for people; its architecture never accounted for limited context windows, hallucination, and the other weaknesses of an AI.

3. More tasks mean more interfaces.
   An atomic tool signature is fixed. A find tool may begin as a name search, but the requests quickly become “find a treasure chest beside a staircase” (neighborhood reasoning), “find a shrub in this visual style” (semantic filtering), or “count the rocks in this area” (aggregation). The same find now hides several fundamentally different computations. One rigid interface cannot span all of them: either the number of tools explodes or the depth of analysis collapses. My testing also found that offering more MCP tools can actually lower completion rate—not because the tools lack capability, but because the model no longer knows which one to choose.

4. There is no validation, so one bad step poisons everything after it.
   Every tool call can fail. Most MCP systems only “provide data”; they do not tell you whether the data is correct. If the agent says there is a pine tree at xxx, should you believe it? Without a mirror in which it can inspect its own work, errors snowball through multi-step reasoning until the whole task drifts off course.

5. It is slow and expensive—often slower than doing the job yourself.
   Every probe crosses a bridge, waits for the editor, and burns tokens. By the time it finishes a lap, I could have clicked through the task myself. A system whose nominal capabilities cannot beat the time and cost of a human using the editor directly has no place in serious production.

MCP is not the only option available to us:

By where the information comes from, those five approaches form three groups:

MCP + CLI acquire information online through engine interfaces. The data is fresh and reflects the current scene, but the interfaces must be maintained, the returned payload can be huge, and every engine upgrade carries a migration cost.

Exported snapshots + OpenUSD reconstruct the scene from offline data. They avoid occupying a running editor, can be versioned, and are convenient for cross-tool workflows, but they are stale and large scenes can produce enormous snapshots.

Direct screenshots acquire information through images + a VLM, much closer to how a person observes a scene. Their semantic intuition is strong and they are flexible across engine versions, but the ceiling is set entirely by the VLM’s visual ability and by the information present in the image.

Only children choose one. I want all of them—combined in pairs.

For the three kinds of information on the vertical axis of the earlier 3 × 3 matrix, I propose three corresponding implementations:

• Visual information ➡️ 2D asset masks: VLM screenshots + exported snapshot text, extending what an image can communicate.
• Spatial information ➡️ GSDL, the General Scene Description Language: a variable-granularity, human-language-level USD + MCP layer that tackles context explosion and weak spatial perception.
• Semantic information ➡️ Vector Atlas, map-wide semantic encoding: Commandlets combined with the two methods above. The implementation has enough difficult bottlenecks to deserve a separate article.

Let us examine each in turn.

4. The 2D Asset-Mask Method

4.1 Why a Screenshot Alone Is Not Enough

For visual-semantic understanding, the most direct and human-like approach is to hand an image to a VLM—or to an LLM augmented with image-understanding capability.

In the ideal future, a large model would read video, understand spatial relationships, and act accordingly. That begins to sound like a world model. Today’s LLMs do not seem to be there yet.

Some products export extra channels such as depth and normal maps to improve a VLM’s spatial perception. But this depends on what the model saw in training. Even before considering the enormous image-token cost, understanding begins to drift when the image is blurry or when the stylized way an Actor is displayed differs from the training distribution. In my view, feeding raw images directly to the model is therefore not an especially good solution.

But what is the root cause of “not understanding”?

Return to the basics of computer vision. Understanding an image means recovering structured semantic information from its pixels: what is here, and what is over there?

Common foundational computer-vision tasks include image classification, object detection, semantic segmentation, and instance segmentation.

In early July, my colleague Fufu from the platform team proposed an image-recognition-plus-raycast approach: use a small model to identify the Actors in a scene, then raycast to retrieve the corresponding Actor. In other words, object recognition followed by instance selection. This inserts a small visual-analysis model between the application and the LLM, letting the LLM actively acquire and segment the contents of the image.

By the way, placing an intermediate layer between a large model and an application is a wonderful agent-design trick. GSDL, which appears later, follows the same logic.

That proposal inspired me.

But it returns us to the same problem: if the model itself distinguishes objects directly from the image, it will sometimes fail, and the resulting analysis is unstable.

So why not reverse the direction? Let the engine report which Actor owns every pixel and output a mask table to the LLM, compensating for the model’s inability to separate every component of an image reliably.

That is my 2D asset-mask method. When the LLM asks for an image from a position, the editor responds. Optionally, instead of returning only the image, it also reports everything in the frame, compresses that visual information into text, and labels what every part of the image represents. The method slashes token cost while providing extremely stable, high-recall instance segmentation.

4.2 Establishing Visual Ownership

The editor already maintains a Hit Proxy rendering and readback mechanism so users can click objects. Its buffer corresponds to the viewport: every hittable pixel maps to a Hit Proxy, which can then be resolved back to an Actor, Component, or another editor object. We can reuse this mechanism and expose pixel ownership to the LLM. Compared with running a separate vision segmentation model, acquiring and parsing this data is cheap.

Once we know every Actor in the image, do we understand the image’s semantics?

Yes. Absolutely.

By a fortunate coincidence, I had just finished an asset-understanding tool—the previous installment in this series.

While building asset understanding, I generated a description for every asset—the left image above—and artists continue editing and improving those descriptions.

The mask statistics on the right reveal, among other things:

1. how much of the image each asset occupies, how it is distributed, and how far away it is;
2. which source asset it comes from, so it can link back to asset semantics;
3. what the instance is called in the scene and which scene parameters it has.

4.3 Edge Cases and Local Vision-Model Support

Careful readers may have noticed that three kinds of “special asset” reported by Hit Proxy need dedicated channels. I handle each separately:

1. Procedural foliage — split it apart and map each foliage Actor back to the semantics of its source asset.
2. Terrain — Hit Proxy reports an entire Landscape Proxy: one name covering a huge swath of pixels. Artists, however, paint multiple user layers into the weightmap—grass, sand, snow, water, and so on. The integration layer therefore follows a dedicated route: world coordinate → proxy-local UV, calculated from proxy scale → weightmap texture sample → ULandscapeLayerInfoObject::GetLayerName(). No visual inference is involved anywhere in the chain, which makes it highly stable.
3. Built-in engine objects such as sky and atmosphere — there are few of them, so I provide handwritten semantics rather than depending on Hit Proxy.

Now we give the VLM the descriptions + image + mask information, organized at several LoD levels, and ask it to generate a description of the frame. Yes, this is a model describing a description—recursion achieved. These LoD levels can be exposed as interface parameters for different levels of detail.

Here, LoD means the granularity of information shown to the model, not mesh geometric detail, although the numbering direction remains consistent.

We can go further by deploying a small vision-language model locally behind the MCP layer. For example, Qwen2.5-VL-7B int4 averages 5.95 GB of VRAM and peaks at 6.65 GB, small enough to run on most artists’ workstations. Deploying Qwen3.8 27B on the studio LAN is not especially difficult either.

Interpreting images and their attached information is a classic perception task and does not require a long chain of reasoning—the sweet spot for a small model. Modern VLMs are pre-trained on large image or image-text corpora and already possess basic visual-semantic ability. This local sidecar lets a cheap, effective, non-multimodal model such as DeepSeek V4 Flash 0731 use our visual-understanding layer as well.

Roughly speaking, a small model is sufficient on one side of the following boundary and begins to struggle on the other:

The exact deployment choice naturally depends on the environment.

Update, August 10: Alibaba recently released Qwen-MM-Plugins, which is very close to my idea. It moves visual understanding into the harness layer and uses multi-resolution processing and frame sampling to understand images and video, essentially completing the sidecar capability I describe here. I will upgrade my approach accordingly.

With this framework, everything in an image can be assigned semantic ownership. Auxiliary objects that Hit Proxy never hits—post-process volumes, lights, and the like—leave no trace in an image mask, so we need another probing dimension.

Enter GSDL.

5. GSDL: The General Scene Description Language

Never heard of it?

Of course not. I made it up. 🐶

5.1 How Do We Understand Space?

The 2D mask method only fills half the gap. It tells the model what is in the image—visual information plus content semantics—but the spatial-relations row in the 3 × 3 matrix remains unsolved.

The editor was built for people. A person can glance at a viewport and immediately feel whether an area is dense or sparse, whether objects are scattered or huddled together. The editor cannot answer those questions directly.

The data we need is not hiding in a single engine field, waiting to be retrieved. It must be measured, aggregated, and inferred from Actors, coordinates, bounding boxes, collision, terrain, rays, and rendering results. The engine does not expose those conclusions as ready-made APIs. A person acquires them intuitively by walking through a scene; an LLM does not.

That is what GSDL does. Through measurement, it cooks and compresses low-level engine state into scene facts that an AI can query, compare, and verify: “there is a mountain at A,” “there is a depression at B,” and so on.

GSDL is also built on MCP, but it does not expose raw engine interfaces directly. A layer of GSDL query functions sits in between, and those functions are the language’s most important rules and conventions. Every query first performs its measurements inside the scene, then converts the result into standardized language for the LLM. In that sense, it can be viewed as an OpenUSD-like textual description language.

But “description language” only names the final output. More precisely, GSDL is a complete methodology for observing, measuring, and describing a scene for AI. One end connects to geometric and rendering facts in the engine; the other connects to asset semantics. Between them, it organizes what exists, how it is distributed, and how the pieces relate into structured language at multiple LoD levels for the LLM.

This diagram shows one complete query loop. MCP is only a thin wrapper: it does not do the work itself; it forwards calls into the engine.

① The model issues an MCP call—describe_region, find_opening, compare, and so on—which arrives at ②, the MCP query interface: the GSDL query layer described above.

② A file bridge carries the request into UE, where scene probes perform the actual measurements and produce raw, uncompressed facts.

③ The spatial kernel clusters those facts and infers relationships. The GSDL encoder then compresses them according to LoD and a character budget. Only then does the model receive the result.

After reading the result, the model reasons about what to measure next and returns to ① with a new question. The query keeps circling through this loop.

In short, GSDL has two main jobs:

1. understand spatial information by organizing discrete objects into density, structure, relations, visibility, and semantic ownership;
2. compress acquired information into concise but rigorously structured semantic statements.

Let us examine them separately.

5.2 How Is Space Measured?

I will visualize one interface so its implementation is easier to understand.

The interface is called find_opening. It tests whether a cavity contains openings. In a cave, for example, it can measure how many entrances exist; in a room, it can determine whether a window is open.

UE is an excellent simulation tool. There is a surprising amount of “measurement” you can do inside it.

The engine already exposes raycasting—LineTraceSingleByChannel in UE—as a native physics query. Walls, rocks, and doorframes are positive space. An opening is the negative space enclosed by them.

So find_opening does not search for Actors whose names contain Door or Archway. It works backward through collision queries: rays that hit geometry trace the walls of a cavity, while groups of rays that escape continuously into the distance indicate an opening. The system then returns to either side of that opening to measure the real frame.

The implementation can be divided into five steps:

1. Find interior points. Lay down candidate samples inside the cavity and keep those that are genuinely enclosed.
2. Choose observation stations. Select several spatially separated interior points as viewpoints.
3. Cast rays. Fire rays around each station and find directions that escape into the distance.
4. Fuse and classify. Combine observations from all stations and distinguish entrances, high windows, and internal passages.
5. Measure exact dimensions. Return to the edges of each opening and measure its real width and height.

The low-level C++ Probe retrieves terrain, instances, collision hits, and ray results from the live Editor. A Python spatial kernel handles candidate sampling, station selection, sector fusion, classification, and precise-measurement scheduling. The LLM no longer receives more than ten thousand raw rays. It receives a compact set of scene facts with type, position, direction, width, height, source station, and evidence grade. It can quote the measurements directly or revisit the supplied camera positions for screenshot verification.

That is how we complete the spatial side of scene understanding.

5.3 How Is Language Compressed?

After the five measurement stages above, we are holding more than ten thousand rays, hundreds of sample points, and over three thousand instances. We cannot hand all that directly to the model—the context window would explode. This is why the earlier Canvas diagram includes a Python data-reduction layer between the raw probes and the LLM.

Compression here does not mean shortening sentences. It is a set of language-design principles:

1. Description is not a data dump. The tool cooks raw instance data into statements: “this region contains 3,242 instances across 200 asset types, organized into 12 clusters; the largest cluster lies to the southeast…” If the text exceeds its budget, it automatically falls back to a coarser LoD. The underlying detail remains available at finer levels.
2. Disclose progressively; summarize first. Descriptions have four levels, LoD0 through LoD3. LoD3 is a one-sentence summary. LoD2 is an asset inventory with statistics. LoD1 expands clusters and spatial relations. LoD0 preserves per-instance detail. The result is also divided into themed channels. By default, the model receives only LoD3: spend a few tokens deciding whether the region deserves attention, then drill down only when necessary—exactly how a person explores a scene, scanning from afar before walking closer.
3. Aggregation is reversible. Compressed detail is not destroyed. LoD1 shows clusters; if the model wants LoD0 for one cluster, expand drills into only that cluster instead of describing the whole region again.
4. Truncation must confess. Every compressed or sampled result says so: truncated (~30% sampled), (3 of 148), or an approximate value prefixed with ~. Every number tells the model whether it came from the full population or a sample.

This is what I meant earlier by “compressing acquired information into rigorously structured semantic expression”: we compress tokens while progressive disclosure preserves access to the information.

Here is the token cost of four output levels for the same describe_region query:

Here is the complete LoD3 result:

1
2
3
4
5
6
7
8
9
10
11

@gsdl v0.1
@project: ScatterTest
@asset_classes: [Foliage, StaticMesh]
@enrichment: []
@frame: m

region(id=R7, bbox=[0.0..200.0, 0.0..200.0, 0.0..18.0]m) {

  # -- summary (LoD3) --
  R7 summary “anchor=watch_tower; 10021 instances/3 classes; density=0.25/m²; orientation=scattered.”.
}

Line by line:

• @gsdl v0.1 — the version declaration.
• @project: ScatterTest — the map or project being observed.
• @asset_classes: [Foliage, StaticMesh] — the broad asset classes present in the region.
• @enrichment: [] — the semantic-enrichment slot, connected to the asset-semantic encoding from Form 1. It is empty in this example.
• @frame: m — the unit contract. Every following length is in meters and every angle is a compass angle, so the model never has to guess whether 92.05 means centimeters or meters.
• region(id=R7, bbox=[0.0..200.0, 0.0..200.0, 0.0..18.0]m) — the region envelope. Every statement inside the braces applies to exactly this bounding box and not an inch beyond it.
• # -- summary (LoD3) -- — a channel heading that identifies the theme of the following block.
• R7 summary ... — the only body sentence and the only sentence form in the entire output: a triple. The subject is R7, the predicate is summary, and the object is a semicolon-separated compressed statement terminated by a period. Predicates come from a fixed vocabulary—located_at, bound, near, density, and so on—rather than free-form prose.

Inside that predicate:

anchor=watch_tower names the landmark. The anchor is selected by salience—visual area × rarity. Among ten thousand six-meter pine trees, only the eighteen-meter watchtower deserves the title. Every later cluster relation hangs from it: pine_c2 of watch_tower or pine_c2 offset (dir=NE, d=43.2m) uses the tower as an origin. 10021 instances/3 classes reports scale, density=0.25/m² reports spacing, and orientation=scattered is a binned orientation-entropy result: ordered placement or disorder.

People give directions in the same order: first a landmark—“near Black Well”—then a relative comparison—“darker than the cave at Flower-Fruit Mountain.” The anchor turns that human habit into grammar.

Two other grammatical elements are not visible in this sample.

The first is precision marking:

• measured values have no prefix: pine_c1.1 pose [92.05, 12.65, 0.0]m; every number was measured;
• estimated values use ~: watch_tower salience ~1.00; salience is a calculated estimate;
• classifications use #: size_dist {M:20, L:10001}#; bins are rule-based judgments rather than direct measurements.

The second is reference resolution. Similar objects are combined into clusters and the long tail is merged into a diffuse field. No matter how many instances exist, the language usually collapses them into only a dozen or so names. If you have a box of pencils, you do not need to say “pencil A, pencil B, pencil C…” Clustering is another form of referential compression.

Anchor, scale, density, and orientation all fit into those 61 tokens. The model can drill into whichever part it cares about.

LoD0 gives each instance one or two lines:

1
2

pine_c1.1      pose  [92.05, 12.65, 0.0]m @ yaw=99.7°.
pine_c1.1      bound [0.85×0.85×6.41]m.

The model calls the appropriate interface itself to retrieve the detail it needs.

Do you remember the 3 × 3 understanding/expression matrix from Chapter 2? I proposed three expression strategies: reference, comparison, and metaphor. Now we can evaluate them.

Reference is exactly what anchors and clusters provide. pine_c2 and “the group northeast of the watchtower” refer to the same thing. offset (dir=NE, d=43.2m) is the formal version of “near Black Well.”

Comparison is quantized in GSDL. The system can provide precise numbers and comparative bands. But bands require quantization and judgment, and judgment introduces instability. That is why comparison received an orange, hard rating in the matrix.

The third strategy, metaphor—“this cave looks like the White Bone Demon’s lair”—is genuinely difficult. It depends partly on the LLM’s own understanding and partly on Vector Atlas, introduced later, which needs artists to teach and annotate scene semantics.

5.4 Self-Evolving Interfaces

The interface above is only one example. Around July 20, 2026, GSDL’s measurable spatial capabilities looked roughly like this:

Those capabilities correspond directly to the spatial relations discussed above. GSDL also includes supporting spatial capabilities:

Before discussing spatial relations, the LLM must first know what exists and which instance a phrase refers to. Only then does it call the deeper reasoning interfaces.

GSDL exposes a growing collection of question-oriented measurement capabilities. The list was not designed all at once; each evaluation round pushed it in a new direction. What you see above is only one snapshot in time. At present there are roughly 35 interfaces, about 20 of which are stable.

I first tried iterating on them in the conventional way. The result was negligible, mainly because iteration was too slow and my own judgment too subjective. I later switched to evaluation-driven development with /Loop: establish a bank of scene questions, have an LLM judge each round from its execution traces and answers, then let those results drive the next interface change. The improvement was dramatic.

My /Loop process has two stages.

For the first stage, I wrote 36 test questions by hand. The earliest bank contained only 11 questions; I later expanded it to 36, manually phrasing every question and annotating every answer. It nearly killed me.

The questions cover more than a dozen categories, including:

• Description and understanding: what is visible in the current viewport, how a district is planned, and how its functions are divided.
• Spatial relations: how far A is from B, in which direction, which is higher, and which blocks the other.
• Anomaly detection: floating objects, intersections, and obviously misplaced assets.
• Passage structure: how many openings a cave has and how wide they are.
• Composite decomposition: which six of fourteen archway assets form one complete gate, or which components make up a fortified enclosure.
• Honest denial: if the scene contains no cars and no neon, can the model say “none” with evidence instead of hallucinating one?

Every question is written in natural language and deliberately avoids naming any tool.

I run the /Loop with two agents. One runs the evaluation: it opens N subagents with clean contexts and asks each to call MCP; a Fable 5 lead agent only schedules work, writes the summary, and produces requirements. The other changes the code: Opus or GLM 5.2 optimizes the interface logic from those requirements.

The MCP-testing subagents use local Qwen3.6 27B and MiniMax M3. SOTA models do achieve higher completion rates, but production-scale MCP usage can become frighteningly expensive. These tools are meant for production, not leaderboards, so they must be adapted to the models that will actually be deployed.

I focus on three hard groups of metrics:

• Delivery: completion rate, accuracy, and honesty.
• Cost: actual token usage, interface call count, and the interface-schema tax.
• Speed: time per question, average latency in milliseconds, maximum latency, and watchdog timeout count. I sometimes also track health and discoverability.

Large and small models differ sharply in some respects, particularly tool use. Small models often forget which interfaces exist. I eventually built a routing layer that the model can call before querying the scene, so it can confirm which tool is appropriate. This trades more routing calls for a higher completion rate:

Here is how the three hard metrics changed, using ten questions under the same protocol on July 21–22:

Within a few days, tokens fell by 54%, calls fell by 64%, and the system reached a perfect score and held it.

I later expanded the question bank. After more than ten additional rounds, the metrics flattened out. Pushing further no longer made much sense; overfitting began to appear.

Time for a stronger dose.

That brought me to the second stage of /Loop development:

I built a passive-update system. When users call MCP tools during their normal work, a Hook records the exact question and the conversation in a background database. Users are also occasionally asked to rate how completely the LLM finished the task.

A scheduled agent turns those interaction records into a capability-requirements table whenever it detects a weak area. At night, after I leave work, an executor agent implements the capability. Another lead agent then launches many subagents to test the MCP changes, summarizes their reports and recommendations, and feeds them back to the executor. The executor iterates again. The loop continues until the new interface stabilizes.

To make this loop work, I even bought two DGX Sparks to run DeepSeek V4 Flash locally. They are chewing grass as I write this—an otherwise pointless display of armament. I hope my Leader sees this paragraph and gives me a raise. ლ(╹◡╹ლ)

The /Loop is still not as complete as I want. I am continuously tuning it, and it contains far more concerns than the sketch suggests: user authorization, data redaction, sandboxing, audit, rollback, and so on. That is another subject I want to cover next time: a self-evolving agent system driven by real user data.

5.5 A GSDL Call in Practice

Here is one real GSDL invocation from a test I ran on July 23, 2026:

“How many passable openings does this cave have, and how wide is each one?”

It neatly illustrates the difference between “reading Actors” and “understanding space.”

Without GSDL—for example, with a conventional UE MCP approach—the LLM searches for assets whose names contain Archway and treats components such as SM_WallArchway_12x3 and SM_WallArchway_3x6 as openings. In my test, those Archway Actors were the walls surrounding the openings, not the openings themselves, so the model misclassified them.

With GSDL:

1
2
3
4
5
6
7

→ nm_status             ← confirm tool status
→ describe_region       ← survey the region
→ find_opening          ← measure openings with rays
→ fly_to                ← fly to the site
→ capture_view          ← verify with a screenshot

→ answer with type, position, width, and evidence grade

Before the find_opening route existed, the model used 47 tool calls and consumed 64,820 returned characters, yet still mistook wall components containing Archway for entrances. After routing was added, it needed only five calls and 15,540 characters to complete both measurement and visual verification.

The model then flew to each candidate opening and captured a verification image. The mask contained 494 pixels belonging to BP_Sky_Sphere, showing that the line of sight passed through the opening into the outside background. That is only visual supporting evidence for the opening’s existence. Whether it is passable and how wide it is still come from collision rays near the ground and precise frame measurements.

In the end, the system classified ten candidate structures into ground-level passable openings or passages, elevated windows, and skylights, then reported the positions and measured widths of the ground openings. It even found an opening I had never noticed. The important point is not “how many Actors named Archway did it find?” It is that the system can distinguish a cave wall, an internal passage, an exterior entrance, and an elevated opening that cannot be traversed directly.

5.6 Open-Sourcing the Project

Thank you for making it this far.

Once this busy period is over and the company-specific material has been separated, I will open-source GSDL on GitHub. I would be delighted if people tried it and sent suggestions; one person’s energy is simply too limited.

The repository placeholder is already here: OpenGSDL. I expect to upload the code in early September. It will contain both the mask method and GSDL, packaged as a multi-version UE MCP plugin that you can extend or use alongside Epic’s official MCP.

6. Vector Atlas: Semantic Encoding for the Entire Map

I built Vector Atlas because every live model query against a map is painfully slow. It has to load a scene, acquire masks, capture screenshots, and so on. In our project, one such operation takes at least 60 seconds. Searching across an entire world multiplies that latency dramatically. Could the process be made much faster?

I also need far more semantic information about the scene. Artists must annotate much of it; I could never label an entire large map alone. Scene semantics—“this is a particular NPC’s house,” for example—also change as production evolves and therefore cannot be hard-coded. I need a dynamic, editable service for collecting scene semantics.

That is why I built Vector Atlas, a semantic encoding of the whole map.

It combines the 2D mask method and GSDL.

Encode every map offline into a semantically searchable table of viewpoints. People can search the entire map by meaning, add viewpoints manually, and edit their semantics. The same structure also becomes cross-session long-term memory for AI: an agent can add a viewpoint, preserve it as a memory, and retrieve it later.

The promise from the beginning—to “memorize” every place and query it in milliseconds—is fulfilled here. It is an engineered trade: storage for performance.

This article is already far too long, and Vector Atlas is itself a very complicated project. How should search be optimized? How should viewpoints be captured? How should it be deployed? How can the system infer an angle it never photographed? Each question deserves serious thought.

So Vector Atlas, together with the real-user-data self-evolving agent system introduced above, will be the subject of the next article. By then, the system should also be more complete.

For now, here is a short tool demo.

7. The Version for Carbon-Based Lifeforms

8. The Version for Silicon-Based Lifeforms—and a Comparison with Epic’s Approach

I prepared 11 questions and had each model use either Epic’s official MCP or my MCP, NeuroMap build 08-14. Every question started from a clean context, and I repeated the evaluation with three models.

Overall, NeuroMap matched or exceeded delivery quality while sharply reducing tool calls and runtime. In the Luna evaluation, Epic’s MCP made 7,919 calls; NeuroMap made only 313, about one twenty-fifth as many, and took only 28% of the total time.

Epic’s MCP exposes general-purpose, atomic editor operations. The model must route repeatedly and inspect objects one by one—almost a traversal of every Actor in the scene. As scene complexity grows, one question can fragment into hundreds or thousands of calls, and round-trip overhead expands rapidly. NeuroMap aggregates region description, anomaly detection, opening analysis, and other spatial computation on the UE side, then compresses the result with GSDL. Across all three models, that produced fewer calls, shorter long tails, and higher delivery rates.

The most striking accuracy result came from Qwen: my MCP raised accuracy from 59.1% to 86.4%. Aggregated, problem-oriented tools are not merely cheaper; they also reduce the burden on a weaker model that would otherwise have to traverse Actors, choose tools, and infer spatial relations by itself.

NeuroMap still has a relatively high schema overhead, and its tool set continues to be consolidated. There is more optimization ahead.

The following is one real workflow. The AI receives only a vague description—“a table with a Go board on top”—then locates it in a large scene, adjusts the camera, and saves a screenshot.

9. Afterword

Have you noticed something? NeuroBrowser—the plugin from Form 1—and NeuroMap—the plugin from this installment—both inject AI capabilities into the engine editor. Their initials are NB and NM; internally, the Chinese pronunciation gave them the nicknames “Your Dad” and “Your Mom.” Together, they feel a little like rebuilding the editor around AI capabilities.

Perhaps this is what the next generation of game editors should look like.

What do you think?

系列导读：「游戏生产AI落地武林秘籍」记录我来时路，都是血泪史。
我们在第一式「手到擒来」里教会了 AI 认识项目里的每一个资产——它知道 shrub_jn_01a 是哪棵江南矮灌木。但这还不够：认识每个兵卒，不等于看得懂整盘棋。 第二式「观局识阵」，要让 AI 看懂美术在地图上摆下的”阵”。

注：每一篇文章几乎纯手工打造，请放心食用。如果你想直接看实现效果，不在乎实现方法，可以越过第三至第六章。

一、缘起：无法理解的场景

第一式我们通过语义编码认识了每个资产，这一回，我们要回答的是：我在哪里？

众所周知，LLM是不理解场景的，即使截图给它，它也看不懂美术摆下的龙门阵：

1. “这个区域大概是什么风格？”
   • AI：幻觉回答一下。
2. “帮我去找场景中一个在楼梯边的宝箱”
   • AI：遍历世界大纲里面的东西，然后一个个地去找。然后哪个在楼梯边？不知道。
3. “给你一张图，这个图上的位置在我大世界的哪里啊？”
   • AI：什么玩意儿？这我咋找？
4. “帮我找到一个我用石头堆成的山洞”
   • AI：@#￥%……&*

本文要解决的就是这些问题（地图理解因为内容有点多故分成两篇文章，此篇是第一篇，讲述理解地图，第二篇会讲全地图的向量检索）。
经过本轮修炼，你的 AI 能像人一样，真正意义上“理解”你的地图，并且将采集过的场景和位置“背”下来；对于未拍摄的视角，则计划通过相邻视角的插值推断加以补全，最终实现大世界地图任意位置的毫秒级查询。

二、如何理解一个场景？

我们来看一下，人类看到一个场景时，是怎么样的？

1. 最先，是看到游戏场景 → “诶，黑黢黢的”（视觉）
2. 然后，感受了其中的空间关系 → “咦，前面有点光”（空间）
3. 最后，理解了场景所表达的含义 → “哦，这里那个白骨精的山洞”（语义）

他会怎么向别人表述这个场景，他说：

我刚刚去了一个场景，从 黑井附近 按E就可以进去，那个地方 比花果山山洞还要黑，我感觉那里应该就是 隐藏关卡boss白骨精的老窝 了。

根据上面的描述，我将理解和表述能力拆分开来：

难度图例：🟢 易 ｜ 🟡 中 ｜ 🟠 难 ｜ 🔴 极难（对 AI 来说既难以理解，又难以准确表达）

纵轴 · 感知层 对应 AI 理解场景的三类能力——

• 「视觉信息」——这片东西看起来怎么样？（风格、疏密、质感、美学印象）。
• 「空间信息」——A 和 B 在空间位置上，是怎么样的？（被谁包住、贴着谁、和谁相交、谁承载谁、谁遮挡谁——拓扑 + 方向 + 距离 + 承载 + 遮挡）。
  其中拓扑关系（RCC-8：DC / EC / PO / EQ / TPP / NTPP…）是定性空间推理里相当成熟的研究领域，可参考 Cohn & Renz 的经典综述 Qualitative Spatial Representation and Reasoning。承载关系与遮挡关系在视觉感知研究里常被单列（如 Biederman 1982 的 Support / Interposition），但在引擎里它们落在同一套几何数据上，故本框架按数据源归并进空间。
• 「语义信息」——A 在场景里是干嘛用的、起什么叙事作用？
  1. 物件含义——这一个物品里面隐含着什么表达？
  2. 组合关系——组合在一起是什么意义（分体论 mereology，比如一堆石头中间掏个口子叫山洞）。
  3. 设计语义——这个场景很破败，是想表达一个正在经历饥荒的村庄。
  4. 共现关系——A 和 B 经常一起出现吗、什么搭配最稳？
     这一层会同时碰到分布语义学、RDF（知识关系表示）和 FrameNet（框架语义）等不同方向。共现本身是从频次 / 向量里“学”出来的，不是从几何里“算”出来的；如果再进一步建模“当前状态如何影响下一个状态”，才进入 Markov（马尔可夫）模型的范围。我把共现作为语义的一部分，因为它表达的是比单体几何更高一层的场景逻辑。这个涉及到一定的表达，有点超纲本文内容。

横轴 · 表述层（能否把学到的转述出来） 对应本文关心的三种表达策略——

• 「指代」对应 deixis（指示语），靠共同注意锚定，不给坐标，也能指向同一个对象；
• 「比较」会用到 vague quantifier / vague language（模糊量词 / 模糊语言），不一定给出精确数字，而是给出方向、程度和隐含基线；
• 「隐喻」对应 conceptual metaphor（概念隐喻 / 跨域映射），借一种共享的文化经验来表达原本很难直接说清的空间、质感或功能。

一个理想的AI场景理解工具，理论上应该能做到上述 3 * 3 坐标里面所有的东西。
另外，一个优秀的agent，理论上还需要具备：长程场景记忆系统，即我们聊过的东西，再次提起还能查得到。
好，接下来我们来讨论实现。

三、别只聊MCP了，它注定是一个无底洞

其实，我们想要的是一个转译层，夹在引擎数据和大语言模型之间，把LLM想要了解的信息，从引擎中抽取，然后转译给LLM。那可能很自然地想到了MCP。
确实，MCP是过去一段时间很火的方案。
然而MCP这事并不是一个好活儿（不能快速地产出立竿见影的效果），它需要大量的资源投入及调优。
（事实上在过去一段时间里，我耗费了很多精力来调试，我甚至整理了一套调优方法论。每天盯着埋点数据，老实说，效果仍然称不上满意）。

为什么？
如果用过一些 UE 的 MCP 工具，就会发现，当前游戏引擎的场景理解部分存在许多问题：

1. 它并不能真正理解我要的场景风格：
   人是有抽象能力的，即我看到一片草地，我能说出这片草地的疏密；它不能“理解”，也不能“总结”。

2. 上下文爆炸，工具数量几何倍数增长：
   现在很多 MCP 的实际做法是将给 UI 的接口开放给 LLM，这件事情就像是给一个剑客发了一把菜刀，能用，但很别扭。UI 是给人用的，从架构层面上一开始就没有考虑到上下文容量、幻觉等 AI 弱势项。

3. 更多任务，就意味着更多接口：
   原子工具的签名是死的，比如 find 这个功能，一开始只用来按名字查资产；可任务很快变成“找一个在楼梯边的宝箱”（要邻近关系判断）、“找这种风格的灌木”（要语义过滤）、“数一下这片区域有多少块石头”（要聚合统计）。同一个 find，背后其实是几种完全不同的运算。一个写死的接口扛不住这种跨度，硬扛就是要么工具数量爆炸，要么分析深度不够。而且，根据我这段时间的测试，我发现提供越多 MCP 接口，反而会得到更差的完成率——问题不在工具能力不足，而是模型根本不知道选择什么工具。

4. 没有校验，错一步后面全错：
   Agent 每调一次工具都可能出错，绝大多数 MCP只管”给数据”，不管”对不对”。它说”xxx有棵松树”，你信不信？没有一面让它照自己的镜子，错误就在多步推理里越滚越大，最后跑偏了你都不知道是哪一步开始歪的。

5. 又慢又贵，不如自己干：
   单次探查要过桥、要等编辑器、要烧 token，等它转完一圈，我自己点几下都弄完了。账面能力再全，单次的时间和费用打不过”人直上手”，那它在严肃工程里就立不住脚。

其实，除了MCP，我们手边还有很多可以利用的方案：

上面这五种方案，按”信息从哪来”可以归成三类：

• MCP + CLI：通过引擎接口在线获取信息。好处是数据新鲜度最高、能实时反映当前场景状态；代价是接口要自己维护、返回数据量大，还要吃引擎版本迭代的成本。
• 导出快照 + OpenUSD：以离线数据还原场景信息。好处是不占引擎运行时、可版本化、便于跨工具读写；代价是有时差、且大场景下快照体积容易爆炸。
• 直读截图：通过图像 + VLM获取信息，更接近人观察场景的方式。好处是语义直觉强、跨版本灵活；坏处是上限完全取决于 VLM 对图像的理解力和图像本身的信息量。

小孩子才做选择，我全都要！我将上面的方案，两两结合。
针对视觉、空间与语义（上文 3 * 3 坐标纵轴坐标），我分别提出了三个应对的方法及技术实现：

• 视觉信息 ➡️ 2D资产掩码方案 （VLM截图 + 导出快照文本，扩展图片的理解力）
• 空间信息 ➡️ GSDL 通用场景描述语言 （颗粒度可变的人类语言级USD + MCP，解决上下文爆炸问题及场景空间感知弱项）
• 语义信息 ➡️ Vector Atlas 全地图语义编码（Commandlet + 上面二者结合后的全地图扩展，在实现上遇到的卡点太多，值得我单独开一篇文章更详细地来聊）

下面展开介绍。

四、2D资产掩码方案

1. 视觉理解为什么只看截图不够

首先来讲视觉信息的语义理解，理论上来说，直接将图片拿给VLM或是图像理解能力扩展后的LLM，是最直接，最接近人类感知的方案。
理想情况下，如果大模型可以直接读懂视频，并看懂空间关系后，执行对应的操作，这听起来像世界模型干的事情。但目前LLM似乎还不是很能达到类似的效果。

所以我看到一些产品会额外导出深度图、法线图等辅助通道，给 VLM 补足空间感知能力，但这通常仰赖模型的训练数据，抛开图片本身的巨量token消耗不谈，一旦当画面变得很模糊，或者画面中的actor带有的风格化显示，和训练数据不一致的时候，就会产生理解的漂移。
所以在我看来直接喂图的方案并不是一个很优的解决方案。

但”看不懂”的根源在哪？
我们可以回到计算机视觉的课程
——理解画面，本质上是理解像素拼出的图像所代表的物体的含义：把像素还原成”这里是什么、那里是什么”的结构化语义信息。

在图像理解领域，常见的基础任务包括图像分类、目标检测、语义分割和实例分割。
中台的同事 Fufu 在七月初提出来的一个图像识别-射线检测的方案：先通过小模型区分出场景中的 actor（物体识别），然后再通过射线检测，去 get 到这个 actor（实例分割）。这个方案通过视觉分析的小模型的中间层，实现LLM主动地去获取画面中的内容。对画面内容进行分割、解析，以达到对画面的理解的能力。
（BTW：在大模型和应用之间叠中间层，这是 agent 设计中一个很棒的 trick，我后面提到的 GSDL 也是基于这个逻辑）

这个方案启发了我。

可是，这又回到了那个问题，如果只是让模型去直接区分图像，会有一定的失败概率，就又会造成分析的不稳定性这个问题的出现。
于是，为何不反着来？引擎主动提供每个位置上的点分别代表什么actor，然后输出一张掩码表给LLM，以此补足LLM无法完整地区分画面的每个组成部分的问题。

这就是我的“2D 资产掩码方案”——LLM提出要在某个位置拍摄图像，编辑器响应，在拍图图像时，（可选）不只是拍摄图像信息，而是一次性给出整个画面的所有内容，将图片信息压缩成文字，并标注画面中每个部分代表的是什么。（该方案不仅token 消耗猛猛降，同时又达到超高物体识别率，稳定性极高的实例分割 Instance Segmentation方案）

2. 如何建立视觉归属

实际上，引擎编辑器为了支持物体点选，会维护一套 Hit Proxy（点选代理）的渲染与读取机制。其缓冲区与视口对应，可命中的像素会关联一个 Hit Proxy，进而还原到 Actor、Component 或其他编辑器对象。我们可以复用这套数据，将像素归属主动提供给 LLM；相比额外运行视觉分割模型，它的获取和解析成本很低。

拿到了画面中的每一个actor后，就可以知道画面的语义了吗？

是的！当然可以。
因为好巧不巧，我前段时间刚好做了一个资产理解工具（上一期）

在做资产理解的时候，我就做了一件事情，我为每个资产都生成了描述（左图，而且这些描述还在被美术不断地更新，优化）。

资产掩码信息统计下来（右图），能够知道：

1. 这个资产在画面中的占比、分布、距离；
2. 它的原始资产是什么（能链接到资产语义）；
3. 它在场景中叫什么名字，在场景中的参数。

等信息。

3. 边缘情况及视觉模型本地支持

细心的读者可能已经注意到——hit proxy 报回来的对象里，会有三类”特殊资产”要走专属通道，我对他们进行了专门处理：

1. 程序化植被（拆开来，按照foliage的actor归属对应的资产语义）；
2. 地形（hit proxy 报回来的是一整座 Landscape proxy（一块名字、一大片像素），可美术其实在 weightmap 里刷了 N 个用户层（草 / 沙 / 雪 / 水……）。这时接合层要走一条专路：世界坐标 → proxy 局部 UV（按 proxy 的 scale 算）→ weightmap 纹理采样 → ULandscapeLayerInfoObject::GetLayerName()——整条链路不会有任何视觉推断，稳定性会极高）
3. 天空/大气等引擎内置对象（因为数量比较少，手写语义，不依赖 hit proxy）。

现在，我们把这些描述 + 画面 + 掩码信息（按不同 LoD 组织）拿给 VLM，让它为这个画面生成一个描述（套娃了属实是），就可以得到最基本的单次截图信息。这些 LoD 可以做成接口参数，以适配不同颗粒度的使用场景。

这里的 LoD 描述的是“给模型看的信息颗粒度”，不是 Mesh 的几何精度，但编号方向保持一致。

想要更进一步，我们甚至可以在 MCP 中本地部署一个小的视觉语言模型（譬如 Qwen2.5-VL-7B int4 VRAM平均占用5.95GB，峰值6.65GB，可以在大部分的美术本地跑起来。或者团队局域网部署qwen3.8 27b，也不是很难的事情）。
图片及信息解释是典型的感知任务（perception），不存在较长的逻辑链路——这正是小模型的甜区。现在的 VLM 普遍经过大规模图像或图文数据预训练，已经具备了基础的视觉语义能力。这样套一个本地部署中间层，可以让Deepseek v4 flash 0731这样便宜又好用的非多模态模型也能支持我们的视觉理解；
至于小模型能做什么、不能做什么，大致可以这样划分：

但这个部署问题就取决于不同环境的具体情况。

（8.10日补：阿里前几天发布了Qwen-MM-Plugins和我的想法很像，将视觉理解这个事情放到Harness层面，通过多层分辨率设计与抽帧方案实现图片和视频的理解，基本上完成了我这个sidecar视觉辅助能力。我在后面也会升级我的方案）

所以基于我的这个框架，画面里的所有东西都能有语义归属。至于 PPV / 灯光等压根不被命中的辅助对象，图像掩码里自然不会留下它们的痕迹，那么就需要另一种probe的维度（有请GSDL）。

五、GSDL（通用场景描述语言）

没听过这个东西是吧。
我自己造的。🐶

1. 如何理解空间？

2D 掩码方案实际上只补上了一半内容——它让模型知道了“画面里有什么”（视觉信息 + 内容语义），而文首那张场景理解能力表格里的“空间关系”依旧是没做的。

引擎编辑器是为人打造的，对于人而言，只要扫一眼视口，就能感觉到这片区域密不密、东西是散开的还是抱团的；引擎编辑器是不能直接回答这些问题的。

我们要的这些数据，并不是引擎某个字段里藏着、可以直接伸手去拿的数据。它要从 Actor、坐标、包围盒、碰撞、地形、射线和渲染结果里重新测量、统计和推导出来，但引擎并没有提供类似的现成接口。这些信息对于人来说，逛一遍场景就能感性地获得，但是对LLM来说不是。
我的 GSDL 干的就是这件事：通过一些测量的手段，把引擎提供的底层状态，烹饪、压缩成 AI 可以查询、比较和复核的场景事实（scene facts，比如A处有一片高山，B处有一个洼地）。

实现上，它也基于MCP，但并不是直接拿引擎的裸接口——中间隔着一层GSDL查询函数作为中转（这层查询函数，就是GSDL最重要的规则和约定）。每次查询都由这层函数先在场景里完成测量，再把结果转换成标准化的语言返还给大语言模型。所以某种意义上，它也可以被看作一种OpenUSD式的文本描述语言。
不过，“描述语言”只是它最终的产出形态。更准确地说，GSDL 是一整套面向 AI 的场景观测、测量与描述方法论：一头衔接着引擎里的几何和渲染事实，一头衔接着资产语义，再把“这里有什么、怎么分布、彼此是什么关系”整理成多档 LoD 的结构化语言描述，交给 LLM。

这张图就是一次完整查询的环路。MCP 在这里只是一层薄封装——它自己不干活，只负责把调用转进引擎：
① 模型发起 MCP 调用（describe_region、find_opening、compare…），先落进 ② 的“MCP 查询接口”——也就是上一段说的那层 GSDL 查询函数；
② 请求经文件桥接进入 UE，由场景探针真正完成测量，吐出的是尚未压缩的原始事实；
③ 空间内核把这些事实聚类、推导出关系，GSDL 编码器再按 LoD 和字符预算把它们压缩成标准化语言，模型这才读到结果；
读完之后，模型思考、决定下一轮该测什么，再带着新的疑问回到 ①——查询就这样一圈圈转下去。

综上，它主要负责两件事情：

1. 理解空间信息（把离散对象组织成密度、结构、关系、可见性和语义归属）
2. 获取的信息压缩（把获取的信息通过一定的规则压缩成规则严谨的语义表述）

下面我分别介绍。

2. 空间是如何被测量的？

我拿其中一个接口做图形化的展示，让你更加明白地了解它的实现逻辑。
下面这个接口叫作find_opening。它主要测试当前的腔体环境中是否存在开口。
比如说在一个山洞中，通过这种算法能测出环境中存在几个洞口、房间有没有开着的窗户之类的应用场景。

其实UE本身是一个很棒的仿真工具，在里面其实可以做很多“测量”的工作。

游戏引擎本身就有射线检测接口——UE 里叫 LineTraceSingleByChannel，是物理引擎原生开放给上层做碰撞查询的能力。墙、岩石和门框都是正空间，洞口恰恰是它们围出来的负空间。
因此find_opening的查询方法，不是通过搜索名字中带 Door 或 Archway 的 Actor，而是反过来使用碰撞查询：命中的射线勾勒洞壁，能够连续逃逸到远处的射线组成开口，再回到开口两侧寻找真实门框。

实现大致可以拆成五步：

1. 找内部点。 在腔体内铺一批采样点，挑出真正处于洞内的。
2. 选观察位。 从中取几个互相错开的点，作为观察位置。
3. 打射线。 从各观察位向四周发射线，找通向远处的方向。
4. 汇总分类。 合并各处观察，区分洞口、高窗与内部通道。
5. 量具体尺寸。 回到洞口边缘细测，给出实际宽高。

底层的 C++ Probe 负责从 Editor 现场取得地形、实例、碰撞命中和射线结果；Python 空间内核负责候选采样、测站选择、扇区融合、类型判断与精测调度。然后将经过测算的结果结论给到LLM，模型拿到的不再是一万多条原始射线，而是一组带有类型、位置、方向、宽高、来源测站和证据等级的场景事实。模型既能直接引用测量值，也能沿工具给出的机位再去截图复核。
于是我们便完成了对场景的空间关系的理解。

3. 语言是如何被压缩的？

上一节的五步测量走完，手里攥着的是一万多条射线、几百个采样点、三千多个实例的原始事实。这些东西没法直接交给模型——上下文会当场爆炸。（所以我在上面的Canvas图表中有一个python数据整理层，来压缩数据信息再给LLM，下面讲的就是在这个python压缩层）

压缩不是把句子改短，它是语言层的一套设计原则：

1. 描述不等于倒数据。 工具把原始实例数据烹饪成陈述——“*此区域 3242 个实例、200 种资产、聚成 12 簇，最大簇在东南...*”(类似于这样的描述)。给模型看的文本超了预算的话，就自动退到更粗一级的 LoD，信息本身仍保留在更细的层级里。
2. 分层披露，先给摘要。 描述分 LoD0–LoD3 四档：LoD3 是一段话摘要，LoD2 是资产清单与统计，LoD1 展开簇和空间关系，LoD0 保留逐实例明细；另有 channels 按主题切块。模型默认只拿 LoD3 摘要：先花小 token 判断值不值得深看，值得再逐层下钻——和人逛场景一个道理，先扫全景，有兴趣才走近。
3. 聚合是可逆的。 压缩掉的信息不是丢了：LoD1 里看到的是簇，要哪个簇的 LoD0 逐实例精度，expand 只钻那一个，不必重描整片。
4. 截断必须自曝。 凡是压过的地方都带记号：truncated (~30% sampled)、(3 of 148)、宽度带 ~——模型读到的每个数字都知道自己是全量还是样本。

这就是第一节说的“把获取的信息压缩成规则严谨的语义表述”：压缩的是 token，通过渐进披露保留信息。

举个例子。同一个区域执行 describe_region，四档输出的token消耗：

LoD3 全文原样贴出来：

1
2
3
4
5
6
7
8
9
10
11

@gsdl v0.1
@project: ScatterTest
@asset_classes: [Foliage, StaticMesh]
@enrichment: []
@frame: m

region(id=R7, bbox=[0.0..200.0, 0.0..200.0, 0.0..18.0]m) {

  # -- summary (LoD3) --
  R7 summary “锚=watch_tower; 10021实例/3类; 密度0.25/m²; 朝向散.”.
}

LoD3 这段全文，逐行每句话在说什么：

• @gsdl v0.1——版本声明；
• @project: ScatterTest——所在地图；
• @asset_classes: [Foliage, StaticMesh]——区域里出现的资产大类；
• @enrichment: []——语义增强槽，接的是第一式的资产语义编码，这片区域没用到；
• @frame: m——单位约定：往后所有长度一律米、角度一律罗盘度，模型不用猜 92.05 是厘米还是米；
• region(id=R7, bbox=[0.0..200.0, 0.0..200.0, 0.0..18.0]m)——区域外壳：花括号里的每条陈述，管辖范围就是这个包围盒，一寸不多一寸不少；
• # -- summary (LoD3) --——通道块标题，标记下面属于哪个主题块；
• R7 summary “锚=watch_tower; 10021实例/3类; 密度0.25/m²; 朝向散.”.——唯一的正文，也是全文唯一的句式：三元组。主语 R7、谓词 summary、宾语是这句分号串联的压缩陈述，行尾句号收束；谓词都来自一张固定词汇表（located_at、bound、near、density……），没有自由发挥的散文。

谓语里面：
锚=watch_tower——锚是这片区域的地标，由显著度（视觉面积×稀有度）选出，一万棵六米高的松树里，就那座 18 米的瞭望塔配得上；此后簇与簇的空间关系都挂在它身上（pine_c2 of watch_tower、pine_c2 offset (dir=NE, d=43.2m) 就是以它为原点的相对表述）。10021实例/3类 是量级，密度0.25/m² 是疏密，朝向散 是朝向熵的档位（摆放的是乱还是规整）。
人指路也是这个顺序——先报地标（“黑井附近”），再给相对关系（“比花果山山洞还黑”）；锚，就是把人类这个习惯写成了语法。

这段里示例还有两个语法元素你看不到。
一个是精度描述：

• 实测值不带符号（pine_c1.1 pose [92.05, 12.65, 0.0]m，每个数都是量出来的）；
• 估计值带 ~（watch_tower salience ~1.00，显著度是算出来的估计）；
• 分类判定带 #（size_dist {M:20, L:10001}#，分桶是规则判的，不是量的）。

另一个是指代方案：

会把相似的东西整合成一个“簇”，碎尾合并成弥散场，再多的实例，最后会整合成十来个名字（即十几种元素）。比如一盒铅笔，我们没必要说，铅笔A、铅笔B…这也是指代上的一种压缩技巧。

总之，锚点、量级、密度、朝向都在这 61 个 token 里——模型会根据自己在意的点往下深究。
LoD0 是一个实例一到两行，长这样：

1
2

pine_c1.1      pose  [92.05, 12.65, 0.0]m @ yaw=99.7°.
pine_c1.1      bound [0.85×0.85×6.41]m.

模型可以自己调用，去获取自己在意的信息。

聊到这里，你还记得我在 “二、如何理解一个场景” 这一段里，提及的理解/表述 的那个 3* 3的表格吗？我提及表述层有我总结的三种表达策略：「指代」「比较」「隐喻」，现在可以验收了：
「指代」——锚和簇干的就是这个：模型说 pine_c2、说“瞭望塔东北那团”，指的是同一个东西；offset (dir=NE, d=43.2m) 就是“黑井附近”的正式写法。
「比较」——在 GSDL 里是档位化的，既可以给出精确的数值，也要能给出对比档位。但是档位就涉及到量化、判断，而一旦出现判断就会不稳定。所以我在 3* 3的表格里给了比较一个🟠（难）。

第三种「隐喻」（比如：这个洞像白骨精的老窝）确实很难，一方面要靠LLM的自我理解能力，另一方面靠文章后面提到的Vector Atlas（全地图语义编码），需要美术去教和标注。

4. 自进化的接口

上面提到的接口只是一个例子。GSDL 当前能测量的空间能力大致长这样（2026 年 7 月 20 日左右的版本）：

表格里面的功能对应着我们上文提到的空间关系。
然后还会包含一些辅助空间功能——

先知道这里有什么、说的是哪个实例，才谈得上空间关系。LLM才会调用具体的理解接口去做事情。

GSDL 对外提供的是一组面向问题、且仍在生长的测量能力。这份清单不是一次性设计出来的，而是被每一轮评测数据推着长出来的，所以上面给你看到的只是它某一个时间切片的样子。
（目前大概有 35 个接口，其中大概20个左右是目前稳定的）

我之前也试过用常规的方式来对它进行迭代。但是效果微乎其微（问题主要是：迭代速度太慢，另外我的个人判断也过于主观），后面我改成了“基于 /Loop 的评测驱动”的方式来优化接口：先建好一批场景测试题，LLM 直接基于每轮的执行数据和结果自己判断自我迭代，效果有了质的飞跃。

我的 /Loop 循环开发大致分为两个阶段：
第一个阶段，我给它写了36道测试题（因为测试题不够，最早是11道题，后来扩展为36道，全是手动提问并人肉标注答案，累死俺了）。

这些测试题涵盖了：

• 描述与理解（当前视口里有什么、一片城区是怎么规划的、功能怎么分区）
• 空间关系：(A 离 B 多远、在哪个方向、谁比谁高、谁挡着谁）
• 异常检测：(有没有悬空、穿模、明显摆错位的东西）
• 通行结构(这个洞穴有几个洞口、分别多宽）
• 复合体拆解（14 个牌坊资产里哪 6 个组成完整牌坊、一座瓮城由哪些构件构成）
• 诚实否认（场景里没有汽车、没有霓虹灯，能否有依据地说“没有”，而不是硬猜一个）等等十几个方向的内容。

所有题目都用自然语言出题、并且刻意不点名任何工具名。

我使用两个agent都跑/Loop循环，一个跑测试（开N个空白上下文的子agent去调用mcp进行提问，Fable5主agent只负责调度，写总结文档，和提需求），一个改代码（opus/GLM5.2 根据前者的需求优化接口逻辑）。
（跑mcp测试的子 agent 用本地 qwen3.6 27b、minimax m3。SOTA 模型的完成率确实更高，但真实场景里一旦大量铺开，MCP 消耗会是个可怕的数字——工具是面向生产的，不是用来跑分的，那就得按生产环境里真正会用的模型来适配）

我主要看三个主要硬性指标：

• 交付能力（完成率、正确率、诚实度) ；
• 成本（真实Token消耗、接口调用次数、接口 schema 税）；
• 速度（单题耗时、均耗时 ms、最大 ms、看门狗超时数），有时候还会关注点其他指标，包括健康度、可能发现性。

根据我的测试来看，大模型和小模型在某些层面确实有很明显的差距，比如体现在对工具的利用能力上：小模型会经常忘记有哪些工具接口。所以我后面甚至特地做了一个路由层，模型在对场景进行查询之前，可以先调用路由层去确认使用哪个工具会更加合适一点，以调用次数换完成率：

三个硬指标的变化是这样的（取 7 月 21–22 日、10 题同口径的数据）：

几天下来 Token 降了 54%，调用降了 64%，正确题数升到满分并守住。
之后，又扩展了题库。再经过十多轮迭代后，指标就进入了一个比较平缓的状态。然后再往下就没有太大的意义了，再往下就会出现过拟合的情况。
那这个时候就要加大药量了。
于是我转向了第二阶段的/loop的开发工作：

我构建了被动更新系统：用户每天调用MCP工具时会触发我的Hook、在后台数据库会记录用户调用工具时候的具体问题，以及对话记录，用户也被要求时不时给LLM的任务完成度打分。
这些交流记录会被一个定时的Agent整理成能力需求表（如果发现比较弱的能力的话）。在夜间我下班后，由一个执行的Agent来负责实现这些能力，然后另外一个Agent会调用大量的子agent，对mcp进行测试，最后由这些子agent的主agent总结测试报表与意见。反馈给执行agent。然后执行再进行迭代…以此循环数次，直到新接口的指标稳定下来。

为了做好这个loop循环，我甚至重金购买了两台DGX Spark来本地跑deepseek v4 flash（吃草中…一些不知所谓的军火展示，希望Leader看到这段话给我涨点工资谢谢ლ(╹◡╹ლ）。

其实现在这个/loop我并没有布置到我满意的程度，它还在不停的调优（这里面有太多的事情，包括用户授权、数据脱敏、沙箱、审计、回滚等一大堆问题）。这也是我下一期想聊的内容之一——基于用户真实数据的自进化的Agent系统。

5. GSDL的调用案例

下面给大家看一下GSDL的一个真实调用案例（我在2026 年 7 月 23 日进行的测试）：

“这个洞穴有几个可以通行的洞口？它们分别有多宽？”

它很能说明“读取 Actor”和“理解空间”的差别。
如果没有GSDL的话（例如 UE 的 MCP 方案），LLM会直接搜索名字中带 Archway 的资产，把 SM_WallArchway_12x3、SM_WallArchway_3x6 等构件当成洞口。在我的实际测试中，这些叫Archway 的 Actor 实际上是围成洞口的墙，不是洞口本身。这会导致误判。

有GSDL后：

1
2
3
4
5
6
7

→ nm_status             ← 确认工具状态
→ describe_region       ← 区域普查
→ find_opening          ← 射线测洞口
→ fly_to                ← 飞抵现场
→ capture_view          ← 截图复核

→ 带类型、位置、宽度与证据等级的回答

同一个问题，加入 find_opening 路由之前，模型用了 47 次工具调用、消耗 64,820 字符，仍然把名字带 Archway 的墙体构件当成洞口；加入路由后，只用了 5 次调用和 15,540 字符，就完成了测量与视觉复核。

模型随后飞到候选洞口前进行截图复核。掩码中出现了 494 个属于 BP_Sky_Sphere 的像素，说明视线确实能够穿过开口看到外部背景；但这只能作为“开口存在”的视觉旁证。它是否能够通行、宽度是多少，仍由贴近地面的碰撞射线与门框精测结果决定。

最终，系统把候选结构分成了10个地面可通行洞口或通道、高位窗口和天窗，并给出了地面开口的位置与实测宽度。它甚至找出了我此前没有注意到的开口。这里真正重要的不是“搜到了多少个叫 Archway 的 Actor”，而是它能够区分洞壁、内部通道、对外洞口和不可直接通行的高位开口。

6. 项目开源

感谢你看到这里。
这个GSDL项目，我这段时间忙完之后，剥离公司项目内容后，会在Github上开源，欢迎大家一来尝试和提建议。一个人的精力确实太有限了。
项目地址暂时先放在这里了：OpenGSDL 大概9月初会上传。里面会包含着掩码方案 + GSDL。会以 UE 多版本 MCP 插件的形式呈现，你可以基于它做二次开发，或者直接用来辅助UE的官方MCP。

六、Vector Atlas（全地图语义编码）

我之所以做这个东西，是因为每一次模型去查询地图里面的东西，其实都是一个非常久的延迟操作（在这个过程中，要去加载场景，获取掩码、截图…），至少对于我们的项目来说，是一个60s+的操作。更别提，要进行全地图大规模的查询，会显著拉长单次任务的完成时间。所以我在想，能不能让这个过程更加的迅速一点。

并且，我还需要更多关于这个场景的语义信息，这些语义是需要美术来标注的，我一个人是几乎不可能完成整个大地图全部场景的标注，并且场景的语义（比如说，这里是某个NPC的家），会随着项目的推进，一直动态变化，这些也不可能靠我硬编码来实现。所以我需要一个动态地、可编辑的、场景语义收集服务。

所以我做了Vector Atlas（全地图语义编码）这么一个东西。
它结合了我们上面提到的 2D掩码 及 GSDL 两种能力为一身。

把我们全部地图离线编码成一张”可语义检索的视角表”，人可以快速地通过语义检索整张地图里面的内容（可人为添加视角，可人为编辑视角语义），同时也作为 AI 跨会话的长期记忆存储能力（AI添加视角、保留为AI长期记忆，AI 可检索）。

开篇我提到的：把每个位置「背」下来、毫秒级查询的承诺，就靠它兑现。用存储，换性能的工程化实现。

因为这篇文章实在太长了，且Vector Atlas这个东西也是一个非常复杂的项目（要考虑到如何优化搜索、如何拍摄、如何部署、如何让AI得到未拍摄的角度…这些都是要思考的东西）。
所以 Vector Atlas 加上（上面提到的）“基于用户真实数据的自进化的Agent系统”（第二期自进化接口），我会留到下一篇文章详细阐述，那时候，我的系统也更加完善了。在这里仅先简单地预告一下。

下面欣赏一下工具的Demo吧。

七、给碳基生物的版本

八、给硅基生物用的版本以及与Epic方案对比

我准备了 11 道测试题，让每个模型分别只使用 Epic 官方 MCP 或我的 MCP（NeuroMap 08-14版），并保证每道题都是全新上下文，进行三个模型的测试。

总体上，交付能力整体持平或领先的情况下，NeuroMap 显著减少了调用次数和任务耗时。其中，Luna 这一轮，Epic MCP一共调用了 7,919 次；NeuroMap 只调用 313 次，约为它的 1/25，总耗时也只有 28%。
Epic MCP 提供的是通用、原子化的编辑器操作，模型需要反复路由并逐个查询对象（几乎是在遍历场景actor）。场景一复杂，一道题就可能被拆成成百上千次调用，这些往返会迅速放大。NeuroMap 把区域描述、异常检测、洞口分析等空间计算放在 UE 侧聚合，再用 GSDL 压缩返回，所以三个模型上都表现出更少的调用、更短的长尾和更高的交付率。

正确率上，最值得注意的是 Qwen这一轮测试，我的MCP从 59.1% 大幅提高到 86.4%。这说明聚合式接口不仅更省，也能明显降低较弱模型自己遍历 Actor、判断工具和归纳空间关系的难度。

不过，NeuroMap 当前的 Schema 开销仍然偏高，工具集合也还在持续收敛，后面还得持续优化。

下面是一次实际工作流：只给 AI 一个模糊描述，让它在大场景中寻找“摆着围棋的桌子”，定位目标、调整镜头并保存截图。

九、后记

不知道你有没有发现一个问题，不管是NeuroBrowser（第一期的插件）还是NeuroMap（这一期插件，二者简称NB和NM，我们内部简称他们为您爸插件和您妈插件），其实都在把 AI 能力渗入引擎编辑器，有点“以AI能力再造编辑器”的感觉？

或许下一代游戏编辑器应该长这样呢？您觉得呢？

Series intro: “The Kung-Fu Manual for Landing AI in Game Production” is a hands-on series for game technical artists (TAs) and tool developers — a record of the road I walked, all of it written in blood and tears.

1. Origin: Don’t Leave AI in the Dark

Our project is a large open-world game with an asset library on the order of 100k items, spanning a dozen-plus major categories — trees, rocks, shrubs, buildings, and more — and each asset also has mobile variants, LODs, Impostors, and sub-resources.

For AI to take part in game production, it first has to “know” what’s in the project. But the reality is —

1
2
3

"Place a low Jiangnan-style shrub"
  → What the AI sees is shrub_a1_01a, shrub_a1_02b
  → Which one is the low one? Which one belongs in this scene? No idea.

Humans aren’t much better than AI here. Artists find assets the same way — digging through folders, guessing at filenames, asking colleagues, retrieving from memory. A 2024 master’s thesis from Aalto University in Finland[^1] documents the same predicament: in HypeHype’s 5,000+ asset library, searching “forest” fails to find an asset whose description says “woods.”

So making assets understandable is the prerequisite for AI to participate in production. But for AI to understand human things, humans have to teach it (annotation). Annotation is time-consuming, and artists have no obligation to help you with it. So I also need an SOP that lets artists annotate assets while they search — trying to spin a virtuous cycle out of that.

2. The Distance Between Assets and Understanding

In fact, whether you’re a human or an AI, understanding an asset involves three different kinds of distance.

1. What is this asset called? (semantic)
2. What does it look like? (visual)
3. What is it for? Where and how is it used? (logical)

BTW: most good search tools can retrieve every character related to an article — even a string that’s just a stray note buried in a document. But you know how the brain works: you might only remember some “punctum” (Roland Barthes, Camera Lucida) tucked in a corner that once moved you, yet you just can’t recall the title of the thing. Every game-engine search box I know of is exactly like this — it can only find filenames, not some blueprint note inside a file, and it doesn’t support more complex search logic.
That’s a product-design problem, and it isn’t “distance” yet. A great search box should be like Everything (yes, I’m praising it by name) — able to search everything in the system! So our search capability should, in theory, be able to fuzzily retrieve an asset from a mere impression. That’s exactly what I’m out to build.

Layer 1: Semantic distance. Literal matching across filenames, paths, and tags — searching “rock” finds rock_granite_01, but searching “石头” or “岩石” (Chinese for “stone”/“rock”) finds nothing, even though the words mean almost the same thing, or differ only by language.

Layer 2: Visual distance. You want to describe a building of a particular form, but — alas, my literature teacher must have passed away too early — you can’t precisely name terms like “庑殿 / 闇栔 / 甍 / 甓” (obscure classical Chinese architectural terms), yet you know it’s that exact thing! The poverty of language turns out to be painfully real.

Layer 3: Logical distance. Searching “a worn-out object you could put in a corner” — this isn’t describing appearance, it’s describing purpose and state. Visual retrieval can’t help here. There’s reasoning baked in: how do you know you can put a broom in the corner of a home, rather than a pig-slaughtering knife? That requires understanding the need and reasoning from it.

Each of the three layers has its blind spots and its strengths, and each suits a different scenario.

3. The Framework: Inner Method

First, an “inner method” of tool design: users are willing to wait for a tool, but the longer they wait, the higher their expectations climb. Waiting a full minute only to get a terrible result is the most unacceptable outcome. So the whole framework is built around layering by response speed — fast results first, slow ones backfilled, never leaving the user hanging.

Following the three “distances” from Section 2, I split retrieval into three layers, each one reaching for a different distance:

• Layer 1 · Base retrieval (fastest, < 100ms): optimizations on top of UE’s native search, doing literal and rule-based matching over filenames / paths / tags. It’s the “safety net” — it works without any AI model.
• Layer 2 · Vector retrieval (instant, < 100ms): this layer actually runs two parallel paths — visual vectors reach for the «visual distance» (search-by-image), and text / description vectors reach for the «semantic distance» (synonyms, Chinese↔English, paraphrases all become searchable, closing the gap that Layer 1’s literal matching can’t cross). Each path recalls independently, then the results are fused and re-ranked.
• Layer 3 · Logical retrieval (re-rank, async in the background): CLI and MCP bring an external LLM into the loop to reach for the hardest «logical distance» — understanding purpose, state, and intent, the kind of query that needs reasoning.

One line to untangle the naming: the three-layer framework maps to the three distances, but the «semantic distance» is closed by a relay between Layer 1 (literal matching) and Layer 2 (text vectors) — which is why, despite being called a “three-layer framework,” it technically runs four paths: «SQL + Visual + Description + LLM». The NeuroBrowser in Section 5 is exactly this inner method put into practice.

4. Sword Summit at Mount Hua · Model Selection

The framework is designed; now for the most critical question: which models?

This isn’t a question you can answer by intuition or by paper leaderboards. There’s a domain gap between game render images and natural images — our thumbnails are rendered in-engine (sky sphere + multi-directional lights + stylized materials), and look nothing like the Flickr / LAION datasets CLIP[^2] was trained on. Leaderboard scores definitely don’t transfer directly.

So I ran two benchmarks: visual-encoder selection %% (maps to subsection 1) %% and description / text-embedding selection %% (maps to subsection 3) %%. The dataset is 100 real project thumbnails + 5 sets of artist-annotated Ground Truth (each set = 1 Chinese description + 3 correct images).

Also, description embedding has no initial data, and making artists annotate it from scratch isn’t quite appropriate either — so I directly used an LLM to generate the initial descriptions for all assets, then ran text-semantic embedding over those descriptions (for future asset ingestion this approach can be reused too, with artists then editing the LLM-generated descriptions). That’s why I also benchmarked a batch of different multimodal LLMs %% (maps to subsection 2) %%.

1. Visual Encoder: 14 Models, Two Rounds of Head-to-Head

Image → semantic vector

1
2
3
4
5
6
7
8
9

flowchart LR
    A["🖼️ Asset thumbnail"] --> B["Visual model"]
    B --> C["1024-dim vector"]
    C --> D["FAISS search"]

    style A fill:#4a90d9,color:#fff
    style B fill:#e8793a,color:#fff
    style C fill:#50b86c,color:#fff
    style D fill:#9b59b6,color:#fff

Round 1 started from the 4 models recommended by the Aalto thesis (a minimal test). The core finding: CLIP without Chinese support completely fails on Chinese queries — Recall@10 of only 13.3%, basically random. The version with an added xlm-roberta multilingual text tower led by a wide margin at 66.7% — a 5–10× gap.

Round 2 expanded along this lead to 10 new models, covering 4 tiers: 2025 SOTA (FG-CLIP 2 family), Chinese-native training (Chinese-CLIP family), multilingual up-and-comers (MEXMA / NLLB / AltCLIP), and the scaled-up family of Round 1’s strongest approach.

The result was unexpected: Qihoo 360’s FG-CLIP 2 large (0.9B params, natively bilingual) took a double crown at P@3=73.3% and R@10=93.3%. In 4 of the 5 queries it nailed all correct answers within the Top-2 — that’s an industrially usable level. A 26.6-point improvement over Round 1’s strongest.

Even more surprising, FG-CLIP 2 large actually beats the larger so400m (1.0B) by 20 points on P@3 — bigger isn’t always better; training steps and adaptation to thumbnail size may matter more.

On bang-for-buck, Chinese-CLIP ViT-L/14@336 ties FG-CLIP 2 large for first on R@10 (93.3%), with a smaller dimension (768 vs 1024), the simplest loading, and the most mature ecosystem — a solid backup.

So I ultimately chose FG-CLIP 2 large as the encoder for image semantics.

⚠️ First, a splash of cold water: all these percentages rest on 5 GT sets × 3 correct images = 15 data points, where a single misplaced result is a ±6.7pp swing. So phrases like “73.3% double crown” and “+26.6pp over Round 1” point in the right direction, but the size of the lead probably has luck baked in — it can’t be treated as settled until GT is expanded to 30+ sets. This warning applies equally to the text benchmark in subsection 3 below — don’t let any single high score go to your head.

2. Benchmarking LLM Generation

Thumbnail → Chinese description

As described above, I need a first wave of initialization text for text semantics, so I planned to use an LLM to generate the initial descriptions for all assets, with artists then editing on top of that text to better fit the “production content” scenario. But understanding an asset and generating a description is no simple matter — different LLMs vary in generation time, quality, and stability. So I ran a test:

1
2
3
4
5
6
7
8
9

flowchart LR
    A["🖼️ Asset thumbnail"] --> B["Multimodal LLM"]
    B --> C["Chinese description text"]
    C --> D["Text embedding"]

    style A fill:#4a90d9,color:#fff
    style B fill:#e8793a,color:#fff
    style C fill:#50b86c,color:#fff
    style D fill:#9b59b6,color:#fff

I pulled 13 mainstream multimodal LLMs (late May 2026) to generate descriptions for the same batch of 100 asset thumbnails (the thumbnail itself takes some care — ideally high-resolution, and if you can, even multi-angle thumbnails; I used a single 2048×2048 thumbnail, and it’s best to avoid over- or under-exposure), benchmarking across four dimensions: cost, speed, quality, stability:

• The benchmark times are for reference only and depend on the API provider too. This is just my environment.
• Also, LLM generation cost depends on the prompt provided. Reproduction costs will differ.
• In testing I used a fixed prompt and ran each image in its own independent context.

A few key findings:

1. The cost gap is staggering — cheapest to priciest is a 20× spread. gemini-3.1-flash-lite ¥0.026/img vs gpt-5.4 ¥0.567/img. For a full 100k-image run, the former is ¥2,607, the latter ¥56,700 — a difference of a dozen-plus RTX 4060 Ti’s worth of money. More surprising, claude-opus is actually cheaper than sonnet (¥0.221 vs ¥0.258) — sonnet got overtaken by opus.

2. Description length varies hugely, but longer isn’t better. Claude-haiku averages the longest at 298 chars with the smallest variance (min 186 ~ max 435), and the best structure (the four-part “shape & structure / material & texture / color & tone / fine details”); qwen3.5-plus is the shortest at just 154 chars. But the later embedding retrieval validated a counterintuitive conclusion — the most human-readable description ≠ the most embedding-friendly description (see next section).

3. Qwen — a dagger hidden in a smile: tag leakage directly pollutes the embedding. qwen3.5-plus leaks <skill> tags (one image output 38 chars of garbage skill\nname visual verdict\n/name\n/skill), and qwen3.6-plus leaks <think chain-of-thought (up to 1,347 chars of reasoning written straight into the output), with stability of only ⭐⭐. These residual markers go straight into the description text and become noise in downstream embedding retrieval. A cleaning step is mandatory before full production (regex tag-stripping + truncation to 800 chars).

4. Kimi-k2.6 is the top domestic pick — 0 anomalies, fastest. 10.2s pure query time, the fastest of all, 0 anomalous outputs, full-star stability. If you’re wary of trust_remote_code or need a fully domestic solution, kimi-k2.6 is the most stable choice. Its discounted version kimi-k2.6-yd occasionally has 1 image with a 2,448-char suspected think leak, usable after adding a max-length filter.

5. The free model is usable but weak. Mimo-v2.5-free is zero-cost but occasionally times out (120s) and has token-loop anomalies (30K–44K stuck in a loop), with some misidentification on water-body types; stability only ⭐⭐⭐. Good for PoC validation, not for full production.

6. Description quality correlates positively with render quality. The higher the render quality, the better the description.

3. Text Embedding

Visual encoding solved the “search-image-find-image” problem, but that search capability is still too shallow. To make AI deeply understand this game’s assets, the more important thing is to break past surface-level image semantics — to grasp who this asset belongs to, and what it means.

And this content is what truly needs artists to teach the AI bit by bit, and it’s also the most “grounded.” So how do we teach it?

My answer is — have the AI first write a description for each asset, then run semantic retrieval over the descriptions (later, artists can edit that description and re-embed it into the vector store). So I invited the previous round’s 4 players to do another cross-benchmark: 7 text-embedding models × 4 description sources = 28 combos, plus an FG-CLIP2 visual baseline + RRF fusion, for 53 experiments total.

Seven embedding candidates (all runnable in bf16 on my 8GB 4060Ti; Qwen3-Embedding-4B/8B excluded because their bf16 weights are ≥8GB):

The four description sources were picked from the previous section’s 13 models: claude-haiku-4-5 (highest human-readability), kimi-k2.6 (production pick), glm-5v-turbo (domestic & stable), gemini-3.1-flash-lite (the value king).

⚠️ gte-Qwen2-1.5B results are unusable, excluded from ranking. Its 2024 custom modeling_qwen.py (bidirectional attention + old KV-cache API) is incompatible with transformers 5.9.0 in three places. I loaded it with native Qwen2Model (causal attention) as a stopgap, but gte’s core is bidirectional — under causal attention, last-token pooling degrades severely (R@10 only 20–60%), which doesn’t represent its true quality. Its C-MTEB of 67.7 suggests its real level should be upper-middle.

The unorthodox blade: the most human-readable description ≠ the most embedding-friendly one

6 fair models × 4 description sources = 24 single-path text retrievals, sorted by R@10 (gte-qwen2 excluded):

⭐ e5-large + gemini single-path R@10=93.3%, tying the FG-CLIP2 image single-path ceiling — the biggest surprise (e5’s C-MTEB is only 58.8, the weakest of the candidates). But this is a single combo over 15 data points, fully within the ±6.7pp noise band — most likely partly luck, and it can’t be taken at face value until GT is expanded.

The most jarring number is in the last row — bge-m3 + claude-haiku has a P@3 of only 26.7%, the worst of all. Yet in the previous section, claude-haiku-4-5 was the “highest human-readability” description model — avg 298 chars, smallest variance, most perfectly structured (the four-part “shape & structure / material & texture / color & tone / fine details”).

Pulling out the average performance of the four description sources across the six embedding models makes it clearer:

Human-Readable Doesn’t Mean Embedding-Friendly

The description ranked #1 for human readability ranks #4 (dead last) for embedding-retrieval P@3; the shortest description, ranked #4 for human readability, ranks #1 for embedding retrieval.

Three likely reasons:

1. Too long → key tokens get truncated. bge-large-zh’s context is only 512 tokens; a 298-char Chinese description likely gets truncated, with the key info lost at the tail.
2. Boilerplate dilutes the semantic keywords. Template words like “shape & structure / material & texture / color & tone / fine details” appear in every description — to the embedding they’re noise: they don’t distinguish asset A from asset B, yet they occupy vector space.
3. CLS pooling is sensitive to long text → the gist vector gets averaged out. The longer the text, the more the CLS token has to encode, which actually dilutes the most critical semantic features (“inn,” “worn-out,” “moss”). gemini-3.1-flash-lite’s descriptions are the shortest (avg 165 chars), letting the embedding focus on the core semantic keywords — less is more.

The Choice of Description Source Matters More Than the Embedding Model

This finding has an important corollary: swapping the description source on the same embedding model can shift P@3 by 20–40 points; swapping the embedding model on the same description usually differs by only ±7pp.

In other words, the description source is the lever, the embedding model is the fine-tune. Investment should go first into iterating the description source (prompt engineering, description cleaning, mixing multiple models’ descriptions); which embedding model you pick barely matters — look at the per-model averages:

None of the 7 Models Pulls Clearly Ahead on Average

All fall within the GT noise band (±7pp). Note that the C-MTEB ranking (stella 68.6 ≫ e5 58.8) is badly inconsistent with the in-domain measurements (e5 ≈ stella ≈ 70%), once again proving “a general benchmark ≠ your domain” — selection must be based on your own in-domain GT.

Fusion: Twin Swords United, 1+1 > 2

The highest single-path text-embedding R@10 reached 93.3% (e5-large + gemini, tying FG-CLIP2), but that’s a single combo over 15 data points, within the noise band — you can’t conclude from it that text has caught up with vision. The more robust judgment: on average, single-path text R@10 sits in the 68–73% range; text can’t replace vision, but it can supplement it. After fusing the visual and text paths with RRF (Reciprocal Rank Fusion, k=60):

Selection Conclusion

In one sentence: Chinese capability is a hard gate, not a bonus; the description model and the retrieval model must be selected separately, with the embedding benchmark as the standard; academic leaderboards are reference-only — you must benchmark on your own data.

📒 While we’re at it, let’s tally the one-time landing costs. Beyond the description generation that keeps coming up (gemini ¥0.026/img, ~¥2,600 for 100k, ~8+ hours), don’t forget a few one-time costs when you actually roll out — full thumbnail export ≈ 33 hours (CPU single-thread, a one-time overnight run, incremental afterward), visual encoding ≈ 3.3 hours, text encoding ≈ 20 minutes (all on a single 4060 Ti). Plus the artists’ later incremental annotation, but that’s optional labor amortized “as you use, you annotate.” All told, the only real money is the bit of API cost for description generation; local encoding is all just electricity — which is why I dared to roll out across 100k-scale assets in one shot.

5. Forging a Handy Weapon: NeuroBrowser (for Carbon-Based Lifeforms)

This weapon is NeuroBrowser — an asset-search panel of mine that runs inside the UE4 editor. It embeds directly as a Slate plugin panel, sitting side by side with the Content Browser (mimicking the native Content Browser experience). An out-of-the-box experience for artists.

Design Philosophy: Three-Layer Retrieval, Progressive Presentation

Back to the three distances raised in [[#2. The Distance Between Assets and Understanding]], NeuroBrowser’s retrieval architecture is also three-layered:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

flowchart TB
    Q["👤 User query"] --> SQL["📋 SQL search<br/>FTS5 + tag filter<br/>&lt; 5ms"]
    Q --> VIS["👁️ Visual search<br/>FG-CLIP-2 large<br/>~30ms"]
    Q --> DESC["📝 Description search<br/>Qwen3-Embedding-0.6B<br/>~30ms"]
    SQL --> RRF["🔄 RRF fusion ranking"]
    VIS --> RRF
    DESC --> RRF
    RRF --> RESULT["⚡ Instant results &lt; 100ms"]
    RRF --> LLM["🤖 LLM Re-rank<br/>300-600ms"]
    LLM --> REFRESH["🔄 Re-rank refresh"]

    style Q fill:#4a90d9,color:#fff
    style RRF fill:#e8793a,color:#fff
    style RESULT fill:#50b86c,color:#fff
    style LLM fill:#9b59b6,color:#fff

Layer 1: SQL search (< 5ms). FTS5 full-text + tag filtering + numeric ranges (tri-count, LOD, material count) + logical combinations (AND/OR/NOT). This layer depends on no AI model and is the safety net — even if the Python backend hasn’t started and no model is loaded, keyword search still works, and it’s 50×+ faster than UE4 native.

Layer 2: Vector search (< 30ms). Visual (FG-CLIP-2) and description (Qwen3-Embedding-0.6B) retrieve independently, each returning Top-K, then fused and ranked via RRF. The three path weights are user-adjustable — for “mossy rock” the visual weight is high, for “worn-out object in a corner” the description weight is high, for “big rock with tri-count < 5000” the SQL weight is high.

Layer 3: LLM re-rank (300–600ms, async in the background). See the next section.

About This Plugin’s Design

1. Heavy local, light server. All indexes (SQLite + FAISS) and models run on the artist’s local machine. Search works fully offline, even with no network. The Python backend (FastAPI) runs on local localhost, depending on no cloud service. The only thing that goes through the cloud is team sync of descriptions/tags — when collaborating, the descriptions you edit are visible to others. Pure-local operation is also supported, so if a user has local data updates, the database can be refreshed quickly.

2. No engine source modification. The whole plugin is built on UE 4.27.2’s public Slate APIs — thumbnails via FAssetThumbnailPool, drag-and-drop via FAssetDragDropOp, asset queries via IAssetRegistry. It never touches the engine’s Private directory and never forks the engine, so future upgrades are painless. It asks little of the engine interface, so it can support more engine versions.

3. Asset-family folding — 100k files collapse to tens of thousands. The same asset often has a PC version + Mobile version (_ios) + LOD variants + material variants (_01a/_01b). If everything is shown flat, the same logical asset appears 5–8 times in the results — that’s not “found it,” that’s “blew up the search.” NeuroBrowser aggregates by “asset family”: _01a/_01b material variants fold into sub-items, _01/_02 different meshes show separately, _ios mobile hides behind the PC version. ~100k asset files → tens of thousands of asset families, results an order of magnitude cleaner.

4. Tag semantics are config-driven, easy to modify. Filenames encode a lot of project conventions — region/scene markers, technical-variant suffixes, and so on — dozens of scene markers plus a dozen-plus technical-variant suffixes, hundreds of rules in total. This logic is itself semantic, and once aligned with the artists it becomes part of the searchable content. Moreover, all these rules are managed via the tag_rules.json config file, not hard-coded. Artists/TAs can add rules themselves, without changing code and recompiling.

5. Descriptions are alive, not dead. The LLM-generated description is only an initial value — artists can edit the description and tags directly in the asset detail panel. After editing, the old embedding is automatically marked stale, and a triple background-update mechanism (lazy + scheduled + manual trigger) rebuilds it asynchronously. Descriptions/tags sync across the team via the cloud — the description you edit is available to a colleague on their next search.

6. A usage-analytics loop. NeuroBrowser has a built-in NeuroBrowserAnalytics module — tab focus, drag-to-viewport, viewport-placement attribution, all logged asynchronously to JSONL. A companion Plotly dashboard shows “which assets are searched most,” “which searches return nothing,” “which assets users drag into scenes most.” This data is the basis for later tuning of search weights and filling in description blind spots.

7. Multiple search modes. Users can adjust each path’s retrieval weight, and refine with logic (filter by tags, filter by model poly-count, etc.).

8. The virtuous cycle — back to the pit we dug at the start. Remember what I said at the start: for AI to understand assets, someone has to teach it; but artists have no obligation to annotate just for you. My solution isn’t “force artists to annotate,” but to embed annotation into the search actions they were going to perform anyway — while searching for and using assets, an artist can casually fix a description or add a tag. These edits sync in real time to the whole team via the cloud shared database: the description you correct today directly benefits a colleague’s search tomorrow. The more it’s used → the more accurate the descriptions/tags → the more accurate the search → the more people want to use it — that’s the “virtuous cycle” promised at the start, and now it’s truly spinning.

Feature Demo

6. Forging a Less-Handy Weapon: MCP & CLI (for Silicon-Based Lifeforms)

Remember why we set out to do this?
We want to teach AI to recognize and understand assets. Now we have image semantics, text descriptions, and a big pile of semantic tags — enough to let AI understand the specific purpose of each asset. We even expose thumbnails to multimodal models, so if it wants to carefully compare two models, it can even spin up a sub-agent to look at the images and compare two assets in detail. (Of course, more exposed interfaces isn’t always better for the LLM — but that’s another story.)

Afterword

To master this kung-fu manual, you must reach the state of “person and weapon as one” with your weapon (AI) — so please download this document and show it to your artificial “idiot” too. Say “get me one of these too,” and you might just get the same tool.

Coming next: Form Two will focus on «how AI understands scenes» (a pit I dug myself — if I can’t pull it off, I might just switch to a different move).

[^1]: FatemiJahromi, S. A. (2024). Enhancing 3D Asset Retrieval with Semantic Search. Aalto University. link
[^2]: Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. OpenAI. CLIP uses contrastive learning to put images and text in a shared vector space — the foundational capability of this approach.

系列导读：「游戏生产AI落地武林秘籍」是一个面向游戏技术美术（TA）和工具开发者的实践系列，记录我来时路，都是我的血泪史。

一、缘起：不让AI两眼一抹黑

我们的项目是一个大型开放世界游戏，资产库规模在 十万级，涵盖乔木、岩石、灌木、建筑等十多个大类，每个资产还有移动端变体、LOD、Impostor、子资源。

要让 AI 参与游戏生产，它首先得”认识”这个项目里有什么。但现实是——

1
2
3

"放一棵江南风格的矮灌木"
  → AI 看到的是 shrub_a1_01a、shrub_a1_02b
  → 哪个是矮的？哪个是应该用在这个场景的？不知道。

人不比 AI 好多少。美术找资产同样靠翻文件夹、猜文件名、问同事，凭借记忆去检索。
芬兰 Aalto 大学 2024 年的硕士论文[^1]记录了同样的困境：HypeHype 平台 5000+ 资产库中，搜”forest”找不到描述里写”woods”的资产；

所以，让资产变得可被理解，是 AI 参与生产的前提。
但，AI 理解人的东西，是需要人来教的（标注）。
这些标注是一个很耗时的事情，美术并没有义务来协助你做这件事情。
所以我还需要一个 SOP，让美术边搜东西，边标注资产。试图以此形成一个正向螺旋。

二、资产与理解的距离

实际上，不管是人还是AI，要理解一个资产，都存在三种不同的距离。

1. 这个资产代号是什么？（语义）
2. 这个资产长什么样？（视觉）
3. 这个资产是干什么用的？用在哪里？怎么用？（逻辑）

BTW：大家用绝大多数优秀的搜索能力，都能检索到和这个文章相关的一切字符，即使这个字符可能只是文档中的一个备注，但是你知道，脑子就是这样，你可能只记得角落里的某个触动你的“刺点（罗兰巴特《明室》）”，可就是想不起这个东西的标题。我所知道的所有游戏引擎的搜索框就是这样，只能搜索到文件名，搜索不到文件里某个蓝图备注，也不支持更复杂的搜索逻辑。
这是产品设计的问题，这还不是“距离”。一个优秀的搜索框，应该像Everything一样（是的，我在实名表扬它），能搜索这个系统里的一切！所以，我们的搜索能力，理论上应该能用印象去模糊地检索到这个资产，这也是我要做的。

第一层：语义距离。 文件名、路径、标签里的字面匹配——搜”rock”能找到 rock_granite_01，但搜”石头”或是“岩石”就搜索不到。即使这几个字的意思几乎很接近，又或只是中英文的区别。

第二层：视觉距离。 想要表达某种特定形式的建筑，但是奈何语文老师去世的早，无法精确地表述出“庑殿”、“闇栔”、“甍”、“甓”这样的词语，但你知道，就是那个东西！此时语言的贫瘠竟是如此的真实。

第三层：逻辑距离。 搜”可以放在角落的破旧物件”——这不是在描述外观，而是在描述用途和状态。视觉检索帮不了这个忙。这里面是包含一定的推理思考，比如你怎么知道这个家里的角落里可以放一个扫帚，而不是放一把杀猪刀，这是需要理解需求，并从需求出发去思考的。

三层各有盲区，也各有强项，需要应用在不同的场景。

三、框架心法

先讲一条工具设计上的”心法”：用户愿意为工具等待，但等得越久、期待就越高。 搜了 1 分钟却给一个糟糕的结果，是用户最不能接受的。所以整个框架是围绕响应速度分层搭起来的——快的先出、慢的后补，绝不让用户干等。

顺着第二节那三种”距离”，我把检索拆成三层，让每一层各去够一种距离：

• 第一层·基础检索（最快，< 100ms）：在 UE 原生检索逻辑上做优化，靠文件名 / 路径 / 标签做字面与规则匹配。它是”保底”——不依赖任何 AI 模型也能用。
• 第二层·向量检索（即时，< 100ms）：这一层其实跑着两条并行的路——视觉向量去够「视觉距离」（搜图找图），文本 / 描述向量去够「语义距离」（同义词、中英、换个说法都搜得到，补上第一层字面匹配跨不过的那道坎）。两路各自召回，再融合排序。
• 第三层·逻辑检索（精排，后台异步）：用 CLI 与 MCP 把外部 LLM 接进来，去够最难的「逻辑距离」——理解用途、状态、意图这类需要推理的查询。

一句话理顺命名：三层框架对应三种距离，但「语义距离」是被第一层（字面匹配）和第二层（文本向量）接力补上的——这也是为什么明明叫”三层框架”，技术上却跑着「SQL + 视觉 + 描述 + LLM」四条路。后面第五节的 NeuroBrowser 就是照这套心法落的地。

四、华山论剑·技术选型

框架设计好了，接下来就是最关键的问题：用什么模型？

这不是一个能靠直觉或论文排行榜回答的问题。游戏渲染图和自然图像之间存在 domain gap——我们的缩略图是引擎内渲染的（天空球 + 多方向光 + 风格化材质），和 CLIP[^2] 训练用的 Flickr / LAION 数据集长得完全不一样。学术排行榜上的分数肯定不能直接迁移。

所以我做了两个 benchmark：视觉编码器选型%% （对应第1小节） %%和描述/文本 Embedding 选型%% （对应第3小节） %%。数据集用的是 100 张项目真实缩略图 + 5 组美术标注的 Ground Truth（每组 1 个中文描述 + 3 张正确图片）。

另外，描述Embedding是不具有初始数据的，这些数据直接让美术去标注也不是非常合适，所以我直接用大语言模型去生成了最初的所有资产的描述，然后再用这个描述去做文本语义的Embedding（未来新资产入库的时候，也可以考虑使用这套方案，然后美术再基于LLM生成的描述数据去修改），所以还对不同的多模态LLM进行了一波评测%% （对应第2小节） %%。

1. 视觉编码器：14 个模型两轮横评

图像 -> 语义向量

1
2
3
4
5
6
7
8
9

flowchart LR
    A["🖼️ 资产缩略图"] --> B["视觉模型"]
    B --> C["1024-dim 向量"]
    C --> D["FAISS 检索"]

    style A fill:#4a90d9,color:#fff
    style B fill:#e8793a,color:#fff
    style C fill:#50b86c,color:#fff
    style D fill:#9b59b6,color:#fff

第一轮从 Aalto 论文推荐的 4 个模型起步（最小测试），核心发现是：不支持中文的 CLIP 面对中文 query 完全失效，Recall@10 只有 13.3%，基本等于随机。而加了 xlm-roberta 多语言文本塔的版本以 66.7% 遥遥领先——5 到 10 倍的差距。

第二轮按这条线索扩展到 10 个新模型，覆盖 4 个 Tier：2025 SOTA（FG-CLIP 2 系列）、中文原生训练（Chinese-CLIP 系列）、多语言新秀（MEXMA / NLLB / AltCLIP）、以及第一轮最强方案的家族放大版。

结果出乎意料：奇虎 360 的 FG-CLIP 2 large（0.9B 参数，双语原生）以 P@3=73.3%、R@10=93.3% 双料夺冠。5 个 query 中有 4 个在 Top-2 就全中正确答案——这是工业可用的水平。比第一轮最强方案提升了 26.6 个百分点。

更意外的是，FG-CLIP 2 large 反而比参数量更大的 so400m（1.0B）在 P@3 上强 20 个百分点——不是越大越好，训练步数和缩略图尺寸适配可能更关键。

性价比方面，Chinese-CLIP ViT-L/14@336 在 R@10 上与 FG-CLIP 2 large 并列第一（93.3%），维度更小（768 vs 1024）、加载最简单、生态最成熟，是稳健备选。

所以我最终选择了 FG-CLIP 2 large 作为图像语义的编码器。

⚠️ 先泼一盆冷水：上面这些百分比都建立在 5 组 GT × 3 张正确图 = 15 个数据点 上，单个结果错位就是 ±6.7pp 的抖动。所以”73.3% 双料夺冠””比第一轮提升 26.6pp”这种话，方向是对的，但领先幅度大概率含运气成分，得等 GT 扩到 30+ 组才能当成定论。这条警告对下面第 3 小节的文本 benchmark 同样成立——别被任何单个高分冲昏头。

2. LLMs生成评测

缩略图 → 中文描述

如上文所述，我需要为文字语义准备第一波初始化文字内容，所以我打算直接用大语言模型去生成了最初的所有资产的描述，然后美术再基于这个文字的基础上进行一些修改，来使其更贴合“生产内容”场景。但是，要对资产进行理解，并生成描述也也不是一个简单的事情，不同LLMs在生成时间、效果、稳定性上都存在差异。为此我也做了一个测试：

1
2
3
4
5
6
7
8
9

flowchart LR
    A["🖼️ 资产缩略图"] --> B["多模态 LLM"]
    B --> C["中文描述文本"]
    C --> D["文本 Embedding"]

    style A fill:#4a90d9,color:#fff
    style B fill:#e8793a,color:#fff
    style C fill:#50b86c,color:#fff
    style D fill:#9b59b6,color:#fff

我拉了 13 个主流多模态 LLM （2026 年 5 月末）对同一批 100 张资产缩略图生成（生成这个缩略图也有讲究，这里最好是一个高分辨，有能力的话，甚至可以考虑多角度缩略图。我这里采用的是2048*2048的单缩略图，且最好不要过曝也不要欠曝）描述，从成本、速度、质量、稳定性四个维度横评：

• 评测的时间仅供参考，和不同API提供商也有关系。这仅仅是我的环境。
• 另外，LLM的生成成本也和提供的提示词有关。复现的成本也会有区别。
• 我在测试的时候使用固定提示词，且每一个图片一个独立地上下文进行的测试。

几个关键发现：

1. 成本差距惊人——最便宜和最贵差 20 倍。 gemini-3.1-flash-lite ¥0.026/张 vs gpt-5.4 ¥0.567/张。按 10 万张全量算，前者 ¥2,607，后者 ¥56,700——差十几张 RTX 4060 Ti 的钱。更意外的是 claude-opus 居然比 sonnet 还便宜（¥0.221 vs ¥0.258），sonnet 竟被 opus 反超。

2. 描述长度差异巨大，但不是越长越好。 Claude-haiku 平均 298 字最长、方差最小（min 186 ~ max 435），结构化最好（”形状结构 / 材质纹理 / 颜色色调 / 细节特征”四段式）；qwen3.5-plus 仅 154 字最短。但后续 Embedding 检索验证了一个反直觉结论——对人类最可读的描述 ≠ 对 Embedding 最友好的描述（详见下节）。

3. 千问——笑里藏刀的暗器：标签泄漏直接污染 Embedding。 qwen3.5-plus 会泄漏 <skill> 标签（某张图输出 38 字垃圾 skill\nname visual verdict\n/name\n/skill），qwen3.6-plus 会泄漏 <think chain-of-thought（最长 1347 字的推理过程直接写进输出），稳定性只有 ⭐⭐。这些残留标记直接进入描述文本，在后续 Embedding 检索中变成噪声。全量生产前必须加清洗步骤（正则去标签 + 截断到 800 字）。

4. Kimi-k2.6 是国产首选——0 异常、速度最快。 10.2s 纯查询时间全场最快，0 异常输出，稳定性满星。如果对 trust_remote_code 有顾虑或需要纯国产方案，kimi-k2.6 是最稳选择。其折扣版 kimi-k2.6-yd 偶发 1 张 2448 字疑似 think 泄漏，加最大字数过滤后可用。

5. 免费模型可用但偏弱。 Mimo-v2.5-free 零成本但偶发 timeout（120s）和 token 循环异常（30K-44K 陷入循环），水体类识别也有失误，稳定性仅 ⭐⭐⭐。适合做 PoC 验证，不适合全量生产。

6. 描述质量与渲染图质量正相关。 渲染图质量越高，描述质量越好。

3. 文本 Embedding

视觉编码解决了”搜图找图”的问题，但这种搜索能力还是太浅层了。
如果要让AI深度理解这款游戏的资产，更重要的是突破表层的图像语义，了解这个资产属于谁？它意味着什么？

而这些内容，是真正地需要美术去一点点教会AI的，并且也是最”落地”的，那怎么去教呢？

我的答案就是——让 AI 先给每个资产写一段描述，再对描述做语义检索（日后美术可以针对这一段描述进行修改，然后重新embedding进向量库）。
所以我请上轮的4个玩家又做了一个交叉评测：7 个文本 Embedding 模型 × 4 个描述源 = 28 种组合，加上 FG-CLIP2 视觉 baseline + RRF 融合，总共 53 组实验。

七个 Embedding 候选（均为我环境里 8GB 4060Ti bf16 跑得动的模型；Qwen3-Embedding-4B/8B 因 bf16 权重 ≥8GB，排除）：

四个描述源从上节 13 个模型中选出：claude-haiku-4-5（人类阅读质量最高）、kimi-k2.6（生产首选）、glm-5v-turbo（国产稳定）、gemini-3.1-flash-lite（性价比之王）。

⚠️ gte-Qwen2-1.5B 结果不可用，已从排名排除。其 2024 版自定义 modeling_qwen.py（bidirectional attention + 旧 KV-cache API）与 transformers 5.9.0 三处不兼容。本次用原生 Qwen2Model（causal attention）应急加载，但 gte 的核心是 bidirectional——causal 下 last-token pooling 严重退化（R@10 仅 20–60%），不代表真实质量。其 C-MTEB 67.7 提示真实水平应是中游偏上。

剑走偏锋：对人类最可读的描述 ≠ 对 Embedding 最友好的描述

6 个公平模型 × 4 描述源 = 24 组单路文本检索，按 R@10 排序（gte-qwen2 排除）：

⭐ e5-large + gemini 单路 R@10=93.3%，追平 FG-CLIP2 图像单路天花板——最大意外（e5 的 C-MTEB 仅 58.8，本是候选里最弱的）。但这是 15 个数据点上的单组合，完全落在 ±6.7pp 噪声带内，大概率部分是运气，必须扩 GT 复核才能当真。

最扎眼的数据在最后一行——bge-m3 + claude-haiku 的 P@3 只有 26.7%，全场最差。
而上节里 claude-haiku-4-5 可是”人类阅读质量最高”的描述模型——avg 298 字、方差最小、结构化最完美（”形状结构 / 材质纹理 / 颜色色调 / 细节特征”四段式）。

把四个描述源在六个 Embedding 模型上的平均表现拉出来看更清楚：

人类可读并不意味着Embedding友好

人类阅读排名第 1 的描述，Embedding 检索 P@3 排名第 4（垫底）；人类阅读排名第 4 的最短描述，Embedding 检索排名第 1。

推测原因有三个：

1. 描述太长 → 关键 token 被截断切掉。 bge-large-zh 的上下文只有 512 token，298 字的中文描述大概率被截断，关键信息丢在尾部。
2. 模板套话稀释了语义关键词。 “形状结构 / 材质纹理 / 颜色色调 / 细节特征”这些模板词在每个描述里都出现，对 Embedding 来说是噪声——它们不区分资产 A 和资产 B，却占据了向量空间。
3. CLS pooling 对长文本敏感 → 主旨向量被均化。 越长的文本，CLS token 需要编码的信息越多，反而把最关键的语义特征（”客栈”、”破旧”、”青苔”）给稀释了。而 gemini-3.1-flash-lite 的描述最短（avg 165 字），反而让 Embedding 聚焦在最核心的语义关键词上——少即是多。

描述源的选择比 Embedding 模型更重要

这个发现有一个重要的推论：同一 Embedding 模型上换描述源，P@3 能差 20~40 个百分点；换 Embedding 模型在同一描述上，差距通常只有 ±7pp。

换句话说，描述源是杠杆，Embedding 模型是微调。投入应优先放在描述源迭代（prompt 工程、描述清洗、混合多模型描述），Embedding 模型选哪个都差距不大——看每模型平均：

7 款模型没有一个在平均意义上明显拉开差距

全部落在 GT 噪声带（±7pp）内。注意 C-MTEB 排名（stella 68.6 ≫ e5 58.8）与本域实测（e5 ≈ stella ≈ 70%）严重不一致，再次印证「通用 benchmark ≠ 你的域」——选型必须以本域 GT 为准。

融合：双剑合璧，1+1 > 2

单路文本 Embedding 的 R@10 最高冲到 93.3%（e5-large + gemini，追平 FG-CLIP2），但这是 15 个数据点上的单组合、落在噪声带内，不可据此认为文本已追平视觉。更稳健的判断是：平均意义上文本单路 R@10 在 68–73% 区间，文本不能替代视觉，但可以补充视觉。
用 RRF（Reciprocal Rank Fusion, k=60）把视觉和文本两路融合后：

选型结论

一句话总结：中文能力是硬门槛不是加分项；描述模型和检索模型必须分开选型，以 Embedding benchmark 为准；学术排行榜只能作参考，必须用自己的数据跑分。

📒 顺便把一次性落地账也算清楚。 除了上面反复出现的描述生成（gemini ¥0.026/张，十万张约 ¥2,600、跑了 8 个多小时），真正铺开还有几笔一次性成本别忘了——缩略图全量导出 ≈ 33 小时（CPU 单线程，一次性夜里跑，后续只走增量）、视觉编码 ≈ 3.3 小时、文本编码 ≈ 20 分钟（都在一张 4060 Ti 上）。再加上美术后续的增量标注，但那是”边用边标”摊薄掉的、可选的人力。算下来真正花钱的就是描述生成那点 API 费，本地编码全是电费——这也是为什么我敢在十万级资产上一次性全量铺开。

五、造一把趁手的兵器：NeuroBrowser（给碳基生物用的）

这把兵器就是 NeuroBrowser——我的一个跑在 UE4 编辑器里的资产搜索面板。它直接嵌入编辑器的 Slate 插件面板，和 Content Browser 并排着（仿照UE原生内容浏览器的体验）。美术开箱即用的体验。

设计哲学：三层检索、渐进呈现

回到 [[#二、资产与理解的距离]] 里提出的三层距离，NeuroBrowser 的检索架构也是三层：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

flowchart TB
    Q["👤 用户查询"] --> SQL["📋 SQL 检索<br/>FTS5 + 标签过滤<br/>&lt; 5ms"]
    Q --> VIS["👁️ 视觉检索<br/>FG-CLIP-2 large<br/>~30ms"]
    Q --> DESC["📝 描述词检索<br/>Qwen3-Embedding-0.6B<br/>~30ms"]
    SQL --> RRF["🔄 RRF 融合排序"]
    VIS --> RRF
    DESC --> RRF
    RRF --> RESULT["⚡ 即时结果 &lt; 100ms"]
    RRF --> LLM["🤖 LLM Re-rank<br/>300-600ms"]
    LLM --> REFRESH["🔄 精排刷新"]

    style Q fill:#4a90d9,color:#fff
    style RRF fill:#e8793a,color:#fff
    style RESULT fill:#50b86c,color:#fff
    style LLM fill:#9b59b6,color:#fff

第一层：SQL 检索（< 5ms）。FTS5 全文检索 + 标签过滤 + 数值范围（三角面、LOD、材质数）+ 逻辑组合（AND/OR/NOT）。这一层不依赖任何 AI 模型，是保底——即使 Python 后端没启动、模型没加载，关键词搜索依然可用，而且比 UE4 原生快 50 倍以上。

第二层：向量检索（< 30ms）。视觉（FG-CLIP-2）和描述词（Qwen3-Embedding-0.6B）两路独立检索，各自返回 Top-K，然后通过 RRF 融合排序。三路权重用户可调——搜”青苔石头”时视觉权重高，搜”角落破旧物件”时描述权重高，搜”三角面 < 5000 的大石头”时 SQL 权重高。

第三层：LLM 精排（300-600ms，后台异步）。见下一个段落。

关于这个插件的设计

1. 重本地、轻服务器。 所有索引（SQLite + FAISS）和模型都跑在美术本地机器上。搜索完全离线可用，断网也能正常工作。Python 后端（FastAPI）跑在本地 localhost，不依赖任何云端服务。唯一走云端的是描述/标签的团队同步——多人协作时，你改的描述别人也能看到。也支持纯本地运行，这样如果用户本地有数据更新，也可以快速更新数据库。

2. 不修改引擎源码。 整个插件基于 UE 4.27.2 的公开 Slate API 实现——缩略图用 FAssetThumbnailPool，拖拽用 FAssetDragDropOp，资产查询用 IAssetRegistry。不碰引擎 Private 目录，不 fork 引擎，后续升级无负担。对引擎接口要求少，可以支持更多引擎版本。

3. 资产族折叠——10 万变 3 万。 同一个资产往往有 PC 版 + Mobile 版（_ios）+ LOD 变体 + 材质变体（_01a/_01b）。如果全部平铺展示，搜索结果里同一个逻辑资产会出现 5-8 条——这不是”搜到了”，这是”搜炸了”。NeuroBrowser 按”资产族”聚合：_01a/_01b 材质变体折叠为子项，_01/_02 不同 Mesh 分开显示，_ios 移动端隐藏在 PC 端后面。十万级资产文件 → 数万个资产族，搜索结果干净一个数量级。

4. 标签语义配置化，方便修改。 文件名里编码了大量的项目约定——用简写区分不同区域 / 场景风格、建筑与资源类别，再配上技术变体后缀（积雪版本、移动端版本等）……30 多个场景标记 + 10 多个技术变体后缀，累计上百种规则，这些逻辑本身是带有语义的，和美术对齐了这些语义后也成为了搜索内容的一部分。另外，这些规则全部通过 tag_rules.json 配置文件管理，不写死在代码里。美术/TA 可以自己加规则，不需要改代码重新编译。

5. 描述是活的，不是死的。 LLM 生成的描述只是初始值——美术可以在资产详情面板里直接修改描述和标签。改完之后，旧的 Embedding 自动标记为 stale，后台三重更新机制（惰性 + 定时 + 手动触发）会异步重建。描述/标签通过云端在团队间同步，你改的描述，同事下次搜索就能用到。

6. 使用分析闭环。 NeuroBrowser 内置了 NeuroBrowserAnalytics 模块——Tab 焦点、拖拽到视口、Viewport 放置归因，全部异步记录到 JSONL。配套的数据分析仪表板可以看”哪些资产被搜得最多”、”哪些搜索没找到结果”、”用户最常拖哪些资产到场景里”。这些数据是后续优化搜索权重、补全描述盲区的依据。

7. 支持多种搜索模式。 用户可调整各路检索权重，支持逻辑精筛选（按标签精筛、按模型面数大小筛选等）。

8. 正向螺旋——回到开篇挖的那个坑。 还记得开篇说的吗：要让 AI 理解资产，得有人来教；但美术没义务专门替你做标注。我的解法不是”逼美术标注”，而是把标注嵌进他们本来就要做的搜索动作里——美术搜资产、用资产的过程中，顺手就能改一条描述、补一个标签。这些改动通过云端共享数据库实时同步给全队：你今天纠正的一条描述，同事明天搜索时就直接受益。用得越多 → 描述/标签越准 → 搜得越准 → 越多人愿意用——这就是开篇许诺的那个”正向螺旋”，现在它真的转起来了。

功能Demo演示

六、再做一把不趁手的兵器：MCP、CLI（给硅基生物用的）

还记得我们做这件事情的初衷吗？
我们要教会 AI 去认识、理解资产。现在我们有了图像语义、文本描述，还有一大堆的语义标签，我们足以让AI理解每一个资产的具体用途，甚至开放了缩略图给多模态，它如果想仔细对比两个模型，甚至可以开一个子Agent去看图仔细对比两个资产。（当然，并不是开放的接口越多，对于LLM越好，而这就是另外一个故事了）。

七、落地之后：把「行为构成」摊开看

兵器造好了，得看它真上了战场是什么成色。第五节里吹过的那个「使用分析闭环」现在真转起来了——NeuroBrowser 自带的埋点把上线以来每一次操作都记成了 JSONL，下面这张「行为构成」就是这段时间的体检报告：横轴是次数，每一根条都是一个动作。

在部署的第一周时间，测试的6 个用户开了 130 个会话，发起 1403 次搜索、选中资产 681 次。从数据上说，NeuroBrowser和UE的Content Browser目前在使用量上几乎平分秋色，虽然UE Content Browser使用数据更好看一点。从使用习惯上看，它没变成「替代品」，而是变成了「发现层」。 原生内容浏览器面板被打开了 1814 次，NeuroBrowser 被打开 1517 次，老的搜索习惯还在；

另外，「真正把资产放进场景」这一步：从原生 Content Browser 放进去 892 个，从 NeuroBrowser 拖进去的只有 118 个，差了将近 8 倍。也就是说——大家在 NeuroBrowser 里「找」，找到之后还是切回原生 CB 去「放」。 它现在更像一把放大镜，还不是那把能单挑的兵器。我觉得这可能是逻辑设计上的问题，因为我还没有完全去模仿UE原生的路径（目前不能完全地看到除了选定资产以外的资产导致的），这个可能后面还要调优。

目前，作为刚刚上线的功能，能有这水平，我个人还是比较满意的。后面再根据数据一点点调整吧。

后记

本武功秘籍，想要学会，必须和你的武器（AI）达成人剑合一的状态，所以请把本文档下载下来，给你的人工智障也看看。 并说“给俺也整一个”，你或许可以获取就同款工具。

预告：第二式将聚焦「AI 如何去理解场景」（自己挖的坑，如果没做出来，可能会换个招式），敬请期待。

[^1]: FatemiJahromi, S. A. (2024). Enhancing 3D Asset Retrieval with Semantic Search. Aalto University. 链接
[^2]: Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. OpenAI. CLIP 通过对比学习让图文共享同一向量空间，是本次方案的基础能力。

Starting point: the asset semantic-search project from [[20260522.SearchToolDesign（Private）]]
Goal: understand the principles of RAG → run a minimal demo against local image data

1. What Is RAG?

RAG = Retrieval-Augmented Generation

In one sentence: first find relevant content from a knowledge base, then use that content to help the AI answer the question.

1.1 Why Do We Need RAG?

2. The Complete RAG Pipeline

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

╔══════════════════════════════════════════════════════════╗
║              Indexing Stage (offline, one-time build)     ║
║                                                          ║
║  Raw data (docs / images / asset descriptions)            ║
║       ↓                                                  ║
║  Embedding model  →  content → fixed-length vectors       ║
║       ↓                                                  ║
║  Vector database (FAISS / ChromaDB) stores vectors,       ║
║  enabling fast retrieval                                 ║
╚══════════════════════════════════════════════════════════╝
                          ↕
╔══════════════════════════════════════════════════════════╗
║              Retrieval Stage (online, per query)          ║
║                                                          ║
║  User question  →  Embedding model  →  question vector    ║
║       ↓                                                  ║
║  Vector DB: compute similarity, find Top-K nearest        ║
║       ↓                                                  ║
║  Return corresponding raw content (doc chunks /           ║
║  image paths / asset info)                               ║
╚══════════════════════════════════════════════════════════╝
                          ↕ (optional)
╔══════════════════════════════════════════════════════════╗
║              Generation Stage (optional, classical RAG)   ║
║                                                          ║
║  [User question] + [retrieved content]  →  LLM  →         ║
║  grounded answer                                         ║
╚══════════════════════════════════════════════════════════╝

3. Core Concept Explanations

3.1 Embedding (Vectorization)

Compress any content (text, images) into a fixed-length list of numbers (a vector), such that:

• Semantically similar content → vectors that are close in space
• Semantically different content → vectors that are far apart in space

1
2
3

"a willow tree"     → [0.12, -0.34, 0.87, ..., 0.05]  (768-dim)
"willows by the river" → [0.11, -0.31, 0.89, ..., 0.06] ← very close!
"a rock"             → [-0.45, 0.72, -0.13, ..., 0.91] ← far away

Common Embedding Models:

3.2 FAISS (Vector Similarity Search Library)

FAISS = Facebook AI Similarity Search (open-sourced by Meta)

Why do we need it?
When you have 20,000 vectors and want to find the Top-10 most similar to a query vector, comparing them one by one needs 20,000 computations. FAISS uses special data structures to speed this up by tens to thousands of times.

An analogy:

• Brute-force search = flip through every book in a library to find the one closest to your interests
• FAISS = the library is partitioned by topic, and you go straight to the “Botany” shelf

Three Main Index Types:

3.3 Similarity Metrics

4. Image RAG (The Special Power of CLIP)

The core innovation of CLIP: make images and text share the same vector space.

1
2
3
4

"a willow tree" (text)  → CLIP text encoder  → vector A
 🌳 (willow image)       → CLIP image encoder → vector B

Vector A ≈ Vector B  ← this is CLIP's training objective!

This means you can:

• text → image: input “give me a flowchart” and find the most similar images in the library
• image → image: upload a reference image to find assets with similar style
• image → text: input an image to find the most relevant descriptive documents

This is exactly the principle behind the “text-to-image search” feature of the asset-search tool.

5. Local Demo Plan

5.1 Goal

Run a minimal image semantic-retrieval demo using the blog images (PNG/JPG) in D:\Project\UGit\MyPicGo\Images\:

• Input: a piece of text description (e.g. “code screenshot”, “flowchart”, “UI interface”)
• Output: paths to the Top-5 most similar images in the library

5.2 Tech Stack

1
2
3
4
5
6
7

D:\Project\UGit\MyPicGo\Images\  (blog images, ~100+ PNGs/JPGs)
    ↓ batch read
CLIP ViT-B/32  →  visual encoder, each image → 512-dim vector
    ↓ build index
FAISS IndexFlatL2  →  in-memory index (no GPU, no server needed)
    ↓ query
user text → CLIP text encoder → 512-dim vector → Top-K retrieval

5.3 Hands-on Process

1. Create project directory D:\Project\UGit\PicGoRAGDemo\ + Python venv
2. Install deps: torch / transformers / faiss-cpu / modelscope / Pillow / numpy
3. Write build_index.py (indexing stage) + search.py (query stage)
4. Run python build_index.py: encode 225 images into 225 × 512-dim vectors, build FAISS index and dump to disk (~15s on CPU)
5. Run python search.py "query" to validate retrieval (single query < 100ms)

Three key pitfalls:

• transformers 5.x changed the CLIP API: get_image_features(...) now returns BaseModelOutputWithPooling (not a bare Tensor anymore); you must call .pooler_output to get the 512-dim vector.
• HuggingFace is unreachable from mainland China: direct connection times out and mirrors are unstable; switch to ModelScope (Alibaba’s ModelScope — domestic servers, no obstacles).
• ModelScope uses different naming: HF’s OFA-Sys/chinese-clip-vit-base-patch16 corresponds to AI-ModelScope/chinese-clip-vit-base-patch16 on ModelScope (AI-ModelScope is the HF-format mirror namespace on ModelScope; the file structure is identical to HF and transformers.AutoModel can load it directly).

5.4 Test Findings: Comparing Chinese vs English Query Effectiveness

Real testing found that English queries are slightly more accurate:

Possible reason: the blog images are mostly English-language technical screenshots (code, terminal, UI text are mostly in English). OpenAI CLIP simultaneously recognizes English text + visual features, so its hits are more reliable; Chinese-CLIP’s training data leans toward Chinese everyday/news content and isn’t sharp enough on the boundaries of technical visual concepts like “flowchart”.

This validates the core point from §6.1: the performance of an embedding model on your domain depends on the training data distribution — choose models based on the scenario.

5.5 Complete Code

build_index.py (indexing stage)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

"""build_index.py — encode N images into N 512-dim vectors and build the FAISS index

Outputs:
  index.faiss  FAISS index (row i = 512-dim vector of the i-th image)
  paths.json   row number → original image path
"""
from __future__ import annotations
import json
from pathlib import Path
import faiss, numpy as np, torch
from PIL import Image
from modelscope import snapshot_download
from transformers import AutoModel, AutoProcessor

# ===== Config =====
IMAGE_DIR  = Path(r"D:\Project\UGit\MyPicGo\Images")
INDEX_PATH = Path(__file__).parent / "index.faiss"
PATHS_PATH = Path(__file__).parent / "paths.json"
# AI-ModelScope is the HF-format mirror namespace on ModelScope, directly reachable in China
MODEL_ID   = "AI-ModelScope/chinese-clip-vit-base-patch16"
BATCH_SIZE = 16
IMG_EXTS   = {".png", ".jpg", ".jpeg", ".webp", ".bmp"}


def collect_images(root: Path) -> list[Path]:
    """Recursively scan all images."""
    return sorted(p for p in root.rglob("*") if p.suffix.lower() in IMG_EXTS)


def main() -> None:
    device = "cuda" if torch.cuda.is_available() else "cpu"

    # === 1. Load model ===
    # snapshot_download downloads ~700MB on first run to ~/.cache/modelscope/,
    # then returns the local path directly
    model_dir = snapshot_download(MODEL_ID)
    model     = AutoModel.from_pretrained(model_dir).to(device).eval()
    processor = AutoProcessor.from_pretrained(model_dir)

    # === 2. Scan image library ===
    paths = collect_images(IMAGE_DIR)
    if not paths:
        raise SystemExit(f"no images found under {IMAGE_DIR}")

    # === 3. Batch encoding ===
    embeddings, ok_paths = [], []
    for i in range(0, len(paths), BATCH_SIZE):
        batch = paths[i:i + BATCH_SIZE]
        imgs, ok = [], []
        for p in batch:
            try:
                imgs.append(Image.open(p).convert("RGB"))
                ok.append(p)
            except Exception as e:
                print(f"  skip {p.name}: {e}")
        if not imgs:
            continue
        inputs = processor(images=imgs, return_tensors="pt").to(device)
        with torch.no_grad():
            # transformers 5.x: get_image_features returns BaseModelOutputWithPooling;
            # the 512-dim projection vector is in the .pooler_output field
            feats = model.get_image_features(**inputs).pooler_output
        # L2 normalize: after normalization, L2 distance ranking ≡ cosine similarity ranking
        feats = feats / feats.norm(dim=-1, keepdim=True)
        embeddings.append(feats.cpu().numpy().astype("float32"))
        ok_paths.extend(ok)

    # === 4. Build FAISS index and dump to disk ===
    mat = np.vstack(embeddings)
    # IndexFlatL2: exact brute-force retrieval; sufficient for < 100K vectors,
    # single-query latency < 1ms
    index = faiss.IndexFlatL2(mat.shape[1])
    index.add(mat)
    faiss.write_index(index, str(INDEX_PATH))
    PATHS_PATH.write_text(
        json.dumps([str(p) for p in ok_paths], ensure_ascii=False, indent=2),
        encoding="utf-8",
    )


if __name__ == "__main__":
    main()

search.py (query stage)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

"""search.py — text query → CLIP text encoding → FAISS Top-K → print paths

Usage:
  python search.py "代码截图"
  python search.py "flowchart" --topk 10
  python search.py "blue UI" --open    # also open the Top-1 result
"""
from __future__ import annotations
import argparse, json, os
from pathlib import Path
import faiss, torch
from modelscope import snapshot_download
from transformers import AutoModel, AutoProcessor

INDEX_PATH = Path(__file__).parent / "index.faiss"
PATHS_PATH = Path(__file__).parent / "paths.json"
# Must match build_index.py — different models' vector spaces are incompatible
MODEL_ID   = "AI-ModelScope/chinese-clip-vit-base-patch16"


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("query", type=str, help="Natural-language query")
    parser.add_argument("--topk", type=int, default=5)
    parser.add_argument("--open", action="store_true", help="Open the Top-1 result")
    args = parser.parse_args()

    if not INDEX_PATH.exists():
        raise SystemExit(f"Cannot find {INDEX_PATH.name}; run python build_index.py first")

    # === 1. Load model + index ===
    device = "cuda" if torch.cuda.is_available() else "cpu"
    model_dir = snapshot_download(MODEL_ID)
    model     = AutoModel.from_pretrained(model_dir).to(device).eval()
    processor = AutoProcessor.from_pretrained(model_dir)
    index = faiss.read_index(str(INDEX_PATH))
    paths = json.loads(PATHS_PATH.read_text(encoding="utf-8"))

    # === 2. Encode text into a vector ===
    inputs = processor(text=[args.query], return_tensors="pt", padding=True).to(device)
    with torch.no_grad():
        # Also take .pooler_output to get the 512-dim projection vector
        feat = model.get_text_features(**inputs).pooler_output
    feat = feat / feat.norm(dim=-1, keepdim=True)
    query_vec = feat.cpu().numpy().astype("float32")

    # === 3. FAISS Top-K retrieval ===
    distances, indices = index.search(query_vec, args.topk)

    # === 4. Print results ===
    for rank, (idx, dist) in enumerate(zip(indices[0], distances[0]), 1):
        # For L2-normalized vectors: cos_sim = 1 - L2_dist^2 / 2
        sim = 1.0 - dist / 2.0
        print(f"  #{rank}  sim={sim:.3f}  {paths[idx]}")

    if args.open and len(indices[0]) > 0:
        os.startfile(paths[indices[0][0]])


if __name__ == "__main__":
    main()

6. Learning Resources

6.1 Introductory Articles (Easy → Deep)

6.2 Must-Read Papers (Optional)

6.3 Chinese Resources

• BAAI BGE series documentation: official docs for Chinese embedding models; directly guides model selection
• Zhihu “RAG Practice”: notes on pitfalls by Chinese engineers; very practical
• Bilibili @跟李沐学AI: deep-learning fundamentals; necessary background for understanding embeddings

7. Notes

Module 1: RAG Overview

1. RAG Architecture

Inside a RAG system, there is first a retriever that has access to the database and fires off a query (similar to a database). The retriever then receives the result of the query (the information deemed most relevant). It then uses this written, possibly most relevant information to generate an augmented prompt.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

# ============================================================
# Stage 1: Define the user question (the original prompt)
# ============================================================
# This is the user's original question, containing time-sensitive
# information ("this weekend") that the large model's own training
# data cannot cover in real time.
prompt = "Why are hotel prices in Vancouver super expensive this weekend?"


# ============================================================
# Stage 2: Naive Generation (baseline for comparison)
# ============================================================
# Throw the original prompt directly to the LLM to generate an answer.
# Downside: the model doesn't know what events are happening in
# Vancouver "this weekend", so it can only offer generic guesses
# (tourist season, exchange rate, etc.) and may hallucinate.
generate(prompt)


# ============================================================
# Stage 3: Retrieve relevant documents
# ============================================================
# Use the prompt to query external knowledge bases (vector DB,
# search engine, internal docs, etc.) and retrieve real-time /
# authoritative information related to the question, such as:
#   - concerts, conventions, sports events happening this weekend
#   - hotel-industry supply/demand news
# These documents are key material for the next "augmentation" step.
retrieved_documents = retrieve(prompt)
print(retrieved_documents)


# ============================================================
# Stage 4: Construct the augmented prompt
# ============================================================
# Stitch "original question" + "retrieved documents" into a new
# prompt, explicitly feeding external knowledge to the model so
# it answers with something to rely on.
augmented_prompt = f"""Respond to the following prompt: {prompt}

using the following retrieved information to help you answer {retrieved_documents}"""


# ============================================================
# Stage 5: Regenerate based on the augmented prompt
# ============================================================
# The LLM now receives "question + contextual evidence"; the
# output is more accurate, more timely, and reduces hallucination.
# This is the complete closed loop of RAG
# (Retrieval-Augmented Generation):
#   Retrieve → Augment → Generate
generate(augmented_prompt)

2. Understanding RAG Through LLMs

LLMs are essentially constantly predicting the probability distribution of the next value to appear; RAG changes that distribution.

3. Information Retrieval: RAG Retrieve vs Search Engine vs Database Query

All three are “looking for things”, but the underlying matching logic is completely different:

A one-line distinction of essence:

1
2
3

Database query — "Are they exactly equal?"   (exact)
Search engine — "Does it have this word?"   (frequency)
RAG retrieve   — "Are they alike in meaning?" (semantic)

Module 2: Information Retrieval & Search Techniques

1. Retriever Architecture Overview

Metadata Filtering

Before or after vector retrieval, use structured fields to hard-filter the candidate set, shrinking the retrieval scope.

• Principle: documents are ingested with metadata fields (e.g. source, date, category); during query, filter first, then rank by vector similarity.
• Advantage: dramatically reduces interference from irrelevant documents, improves both precision and retrieval efficiency.
• Limitation: relies on metadata quality at ingestion time; missing or misclassified fields can wrongly filter out effective documents. It doesn’t understand content and is almost never used alone.

Keyword Searching

Searches documents by exact-word matching; the most classical search method. Mainly includes TF-IDF and BM25.

• Principle: split the query into tokens, compute term frequency (e.g. BM25), return documents containing those words.
• Advantage: fast, results explainable, accurate hits on proper nouns (codes, IDs, model numbers).
• Limitation: doesn’t understand synonyms or paraphrases — say it differently and you might not find it.

Semantic Searching

Encode text into vectors and compute vector similarity to match “similarly-meaningful” content.

• Principle: use an Embedding model to map both query and documents into a high-dimensional vector space; measure distance using cosine similarity, etc.
• Advantage: captures semantic generalization; finds related content even when phrased differently.
• Limitation: higher compute cost; less reliable than keyword search for exact terms (e.g. specific IDs).

2. Two Common Keyword-Retrieval Algorithms in Detail

TF-IDF

Core formula: Score = TF(term, doc) × log(total docs / docs containing term)

• TF (term frequency): the more a term appears in the current doc, the more relevant
• IDF (inverse document frequency): the more docs contain a term, the more “filler” it is — weight drops; rarer terms get higher weight
• Result: high-frequency function words like “the”, “is” score near zero; proper nouns like “quantum entanglement” or “BM25” score high

BM25 (the industry-grade improvement of TF-IDF; Elasticsearch’s default algorithm)

Core improvements:

k1: controls how fast term frequency saturates; higher means “saying it more makes it more important”; lower means “3 times and 100 times are nearly the same”.
b: controls length penalty strength; b=0 ignores length, b=1 strictly scores by density; 0.75 is an industry empirical default.

3. Similarities and Differences Between Semantic Search and Keyword Search

1. Common ground: Digitization

• Prompt and documents each get a vector:
  No matter the search type, the computer can’t directly “read” text. The first step is always turning both the user’s question and the documents in the database into a list of numbers, called a vector.
• Vectors compared to generate scores:
  Once converted into numbers, the computer can use math formulas (e.g. cosine similarity) to compute the “distance” between two vectors. The closer, the higher the score, the more relevant.

2. Core difference: How is the vector generated?

Keyword Search: Count word occurrences

• Principle: the result is called a sparse vector.
• Logic: each position of the vector represents a specific word. If the document contains that word, the corresponding position gets a score (based on term frequency, e.g. BM25).
• Characteristic: literal matching; only recognizes “identical-looking” words.
  Limitation: searching for “doctor” won’t find a document containing “physician” but not “doctor”, because it doesn’t understand word meaning.

Semantic Search: Use an embedding model

• Principle: the result is called a dense vector.
• Logic: instead of counting words directly, text is fed into a pre-trained deep learning model (Embedding Model, e.g. BERT). The model “maps” the text into a multidimensional semantic space.
• Characteristic: understands meaning; the numbers in the vector represent abstract “features” or “concepts”.
  Advantage: recognizes synonyms; even if words don’t match, as long as the “meaning” is close (e.g. “cat” and “kitten”, “doctor” and “physician”), their vectors are mathematically close in space.

3. RRF Algorithm (Balancing Keyword and Semantic)

It is the core technique in hybrid retrieval. When you run both “keyword retrieval” and “semantic retrieval” in your RAG system, you get two completely different scoring lists. RRF (Reciprocal Rank Fusion) merges these two lists fairly into one final ranking list.

1. Core problem: The “apples and oranges” comparison dilemma

Hybrid retrieval faces a fundamental difficulty:

• Keyword (BM25) scores may be 15.4, 12.8, etc.
• Semantic (vector-search) scores (cosine similarity) typically range between 0.8 and 0.9.
• Problem: these two scores have different units and cannot be directly summed or compared.

RRF’s solution: completely ignore raw scores; only look at documents’ positions (ranks) in the lists.

2. Core mechanism

• Reward “consensus” documents: if a document ranks high in both keyword search and semantic search, RRF gives it an extremely high final score.
• Normalize weights across searches: provides a fair comparison across strategies; neither algorithm can dominate just because its score range is larger.
• Score = reciprocal of rank (the origin of the algorithm’s name):
  • Rank 1 → 1/1 = 1.0 score
  • Rank 2 → 1/2 = 0.5 score
  • Rank 10 → 1/10 = 0.1 score
  • Logic: higher rank = higher score; the score drops faster as rank drops.
• Aggregate scores: sum each document’s reciprocal score across all lists; the one with the highest total wins the final ranking.

3. Formula

$$RRF(d) = \sum_{i=1}^{n} \frac{1}{k + rank_i(d)}$$

• rank_i: document d‘s ranking in the i-th retrieval list (starting from 1).
• k: smoothing constant, the industry default is usually 60.

Why is k needed?

Without k, the gap between rank 1 (1.0 score) and rank 100 (0.01 score) is enormous, giving the first-ranked document overwhelming power. k acts as a “shock absorber” — compressing extreme differences and weakening the influence of noisy documents that happen to land at rank 1.

4. RRF’s core advantage: Only cares about rank

• No score normalization needed: no need to convert BM25’s 20 score and vector search’s 0.9 score — compare positions directly.
• Seamless cross-strategy merging: with 2 or 5 different retrieval techniques, as long as each gives a rank list, RRF can fuse fairly.

5. Parameters for tuning semantic vs keyword weights

Standard RRF itself has no direct “semantic vs keyword” weight parameter — k is just a smoothing constant, not a control over their ratio.
But Weighted RRF introduces an independent weight w for each retrieval strategy in the formula:

$$RRF(d) = \sum_{i=1}^{n} \frac{w_i}{k + rank_i(d)}$$

• Increase w_semantic → results skew toward semantic understanding (synonyms, conceptual relevance)
• Increase w_keyword → results skew toward exact literal matching

Where do you tune this weight in an actual implementation?
Take LangChain‘s EnsembleRetriever as the most intuitive example:

1
2
3
4
5
6
7
8
9
10
11
12

from langchain.retrievers import BM25Retriever, EnsembleRetriever
from langchain_community.vectorstores import FAISS

# Two retrievers
bm25_retriever = BM25Retriever.from_documents(docs)
vector_retriever = FAISS.from_documents(docs, embedding_model).as_retriever()

# weights correspond to w_i in the formula; the two values usually sum to 1.0
ensemble_retriever = EnsembleRetriever(
    retrievers=[bm25_retriever, vector_retriever],
    weights=[0.3, 0.7]  # keyword 30%, semantic 70%
)

weights=[0.3, 0.7] are w_keyword and w_semantic from the formula. Increase semantic → better at understanding intent; increase keyword → better at exact proper-noun matching.

4. Evaluation Metrics

Quantifying how good a retriever is with numbers is the scientific basis of system tuning.

Three core metrics

A concrete example to understand Precision and Recall

Scenario: the knowledge base has 100 docs, of which 10 are truly relevant (Ground Truth). The retriever returns 8, of which 6 are truly relevant.

$$Precision = \frac{\text{returned and relevant}}{\text{total returned}} = \frac{6}{8} = 75%$$

$$Recall = \frac{\text{returned and relevant}}{\text{all relevant docs in the library}} = \frac{6}{10} = 60%$$

Precision: from the perspective of “the docs you returned” — how many are real, how much is noise?
Recall: from the perspective of “the knowledge base” — of the 10 correct answers, how many did you find?

The intrinsic tension between Precision and Recall

The two naturally pull against each other:

Practical uses of metrics

• Establish baseline performance: score your current system and make clear where you are
• Validate optimization impact: when changing Embedding model, adjusting Chunk size, or modifying hybrid-retrieval weights, compare before/after metrics to confirm whether the change actually worked

The most critical prerequisite: Ground Truth

All metrics depend on a “ground truth” dataset

• What is Ground Truth? A human-annotated dataset. For example, for question A, pre-annotate that “doc 1” and “doc 5” are the only correct answers in the knowledge base.
• Why does it matter? Recall, Precision, MAP, MRR all require comparing “system’s answers” against “standard answers”. Without Ground Truth, scientific tuning is impossible.

5. Embedding Model In-Depth

Contrastive Training Process

Training objective: pull vectors of similar content closer, push vectors of dissimilar content apart.

Positive Samples: From “Natural Pairings on the Internet”

Humans’ natural behavior on the internet inherently produces massive paired data:

Core idea: the internet itself is a huge “implicit-annotation dataset”. When humans create content in daily life, they are unconsciously labeling data for AI.

Negative Samples: Automatically Generated by Algorithms

Rare Scenarios That Need Human Annotation

Only when you need a high-precision Benchmark (used to test how good a model is) do you bring in human annotation:

1
2
3

STS (Semantic Textual Similarity) dataset:
  Humans score the similarity of two sentences from 1 to 5
  But this is used to "evaluate" the model, not the main data used to "train" it

Training process: forward pass generates vectors → build similarity matrix → InfoNCE loss penalizes cases where “positive samples don’t rank high” → backprop updates weights.

CLIP’s 400M training pairs are almost entirely auto-acquired. The internet itself is the implicit-annotation dataset; human annotation is only used for evaluation benchmarks, not as the main training data.

I also feel that this training process is quite similar to the node graph on my personal website — the node graph has repulsive and attractive forces, the entire node-graph system needs to maintain a stable state. Therefore, to maintain the balance of repulsion and attraction, it needs an adjustment process. And I think this adjustment process may be exactly what embedding is.

A common misconception — LLM parameters vs Embedding dimensions: parameters are the total weights inside the neural network (CLIP has ~150M), i.e. the knowledge learned during training; the dimension is the length of the output vector (e.g. 512), a fixed spec at design time. Parameters are the total knowledge the model learned (the more, the “smarter”); the dimension is the length of the output vector (it bounds the upper limit of expressiveness). They are not the same thing.

Module 3: Information Retrieval with Vector Data

1. From Brute-Force Search to ANN

All vector-retrieval algorithms share one key variable — K: the number of vectors most similar to the query (Top-K) to return. Around “how to efficiently find these K vectors”, two major schools have evolved: exact retrieval and approximate retrieval (ANN).

The most basic algorithm: brute-force search (Brute Force / Flat Search / Linear Scan)

The simplest direct approach — compute the distance (cosine / Euclidean) between the query vector and every vector in the database, then sort and take the Top-K.

In FAISS it’s called IndexFlatL2; in academia it’s called Exact KNN; in engineering it’s often called Brute Force or Linear Scan. They’re all the same thing: build no index, brute force to the end.

• Pros: trivial to implement (one line of numpy); Recall = 100%, the accuracy ceiling and evaluation baseline of all ANN algorithms
• Fatal problem: complexity is O(N×D) (N = number of vectors, D = dimension). The bigger the database, the slower a single query — millions of vectors is already hard; billions are outright unusable
• Suitable for: small datasets (< 100K), offline evaluation, providing ground truth for ANN

The solution: ANN (Approximate Nearest Neighbor) search

Core idea: pre-build an index, skip most vectors that can’t possibly be relevant, only do fine-grained comparison in a small range — sacrificing a tiny bit of accuracy in exchange for orders-of-magnitude faster queries.

Mainstream implementations:

• HNSW (Hierarchical Navigable Small World graphs): based on “skip-list + small-world networks”; complexity drops to O(log N); high accuracy and fast queries — industry mainstream
• IVF (Inverted File Index): first K-means clustering; at query time only scan the nearest few clusters — suited for very large scale
• PQ (Product Quantization): splits high-dim vectors into segments then replaces with cluster codes — compresses vectors 64× or more, dramatically saving memory

In engineering you must trade off between accuracy (Recall) ↔ speed (QPS) ↔ memory; common combinations are IVF+PQ, HNSW+PQ.

2. Vector Databases

A vector database is a database system designed specifically for storing, managing, and retrieving high-dimensional vectors. Ordinary databases store structured rows and columns; vector databases store Embedding vectors — and have ANN indexes built in, making semantic retrieval a first-class citizen.

Core differences from traditional databases

Vector databases usually store vectors + metadata together, supporting “first hard-filter by metadata, then do vector retrieval” — i.e. the combination of Metadata Filtering and semantic retrieval mentioned earlier.

Mainstream Vector Database Comparison

• Qdrant: written in Rust; strong performance; REST/gRPC API; supports payload filtering; open source, self-hostable
• Milvus: designed for very large scale (billions); cloud-native architecture; suited for production distributed scenarios
• ChromaDB: lightest, starts in a few lines of code; first pick for dev/debugging, not suited for large-scale production
• pgvector: PostgreSQL extension; add vector retrieval to an existing PG database with no new system
• FAISS: strictly speaking a library, not a database — no persistence, no CRUD, but the source of all vector DB algorithms underneath

Basic Process for Creating a Vector Database

Step 1 — Database Setup
Create a collection and define its schema — specifying which fields to store, the vector dimension, and which distance metric to use (cosine / L2 / inner product). This is the container for all subsequent operations, equivalent to creating a table.

Step 2 — Loading Documents
Read raw data (text, PDFs, images, etc.) into memory, chunking as needed — splitting long documents into small chunks suitable for embedding, to avoid semantic dilution when a chunk is too long.

Step 3 — Sparse Vectors (for keyword retrieval)
Use algorithms like BM25 to generate sparse vectors for each document chunk — most positions are zero; only positions for terms that appear have weights. Designed for exact keyword matching; this is the keyword side of hybrid retrieval.

Step 4 — Dense Vectors (for semantic retrieval)
Pass document chunks through an Embedding model (e.g. BGE, CLIP), outputting a dense vector for each chunk — every dimension has a value, carrying semantic information. The core of semantic search; understands synonyms and intent.

Step 5 — Build the HNSW Index
Build an HNSW (Hierarchical Navigable Small World) index on top of the dense vectors. Pre-weave a “navigation network” among the vectors; at query time you follow graph jumps to locate, reducing complexity from O(N) to O(log N) and achieving millisecond-level ANN search. This step can be skipped; if skipped, you have brute-force retrieval.

3. Chunk (Chunking Techniques)

Splitting long documents into small chunks suitable for embedding is the key pre-processing step of the RAG indexing stage. Too-large a chunk dilutes semantics; too-small a chunk loses context; whether you split well directly affects retrieval quality.

Why do we need chunking?

Embedding models have a token limit (e.g. BERT series 512 tokens, BGE-M3 8192 tokens). Stuffing an entire book in only yields one fuzzy “average semantic”, and retrieval will struggle to hit specific passages. After chunking, each chunk has its own independent vector, and retrieval precision improves dramatically.

Main chunking strategies

• Fixed-size Chunking: hard-cut by character or token count; crude. Downside: may cut mid-sentence and lose context. Usually paired with Overlap — adjacent chunks share some tokens — to mitigate the truncation issue.
• Semantic Chunking: cuts along sentence boundaries, paragraphs, heading hierarchy; preserves natural semantic integrity. Suited for structured documents (Markdown, PDF).
• Recursive Character Splitting: LangChain’s default strategy; tries priorities like \n\n → \n → period → space in order, cutting on natural boundaries as much as possible; balances simplicity with semantic completeness.

4. Some More Advanced Chunking Techniques

Using LLMs for Semantic Chunking

Traditional chunking relies on rules (paragraphs, headings, fixed size); “LLM semantic chunking” lets the model truly understand the semantic boundaries of content before deciding how to cut — higher quality but also higher cost.

① Embedding-similarity-based Semantic Chunking (SemanticChunker)

Principle: split text into sentences first → compute embedding for each sentence → compute cosine similarity of adjacent sentences → when similarity drops sharply, the topic has shifted; cut there.

1
2
3

sentence1 → sentence2 → sentence3 ↘↘sharp drop↘↘ sentence4 → sentence5
                                          ↑
                                      cut point (semantic break)

• Tool: LangChain’s SemanticChunker directly wraps this logic
• Pros: no LLM inference needed; low cost; truly cuts by semantic breakpoints, not by character count
• Cons: needs a similarity threshold; a badly-chosen threshold cuts too finely or too coarsely

② Proposition Chunking

Principle: use an LLM to further distill each text segment into several atomic propositions — each proposition is an independent, complete, self-contained minimal fact unit.

Example: original text “Einstein published the special theory of relativity in 1905 and won the Nobel Prize for the photoelectric effect”
→ Proposition 1: “Einstein published the special theory of relativity in 1905”
→ Proposition 2: “Einstein won the Nobel Prize for his research on the photoelectric effect”

• Pros: at retrieval time, query and proposition map one-to-one; precision is extremely high; each proposition is self-contained, understandable without context
• Cons: every chunk must invoke an LLM to distill — token-heavy and slow; indexing is expensive
• Suited for: scenarios that demand the highest knowledge-base retrieval quality (medical, legal, precision Q&A)

③ Agentic Chunking

Hand the whole document directly to the LLM, letting it autonomously decide semantic boundaries, output split points, or output the chunks directly. Most flexible; can handle complex unstructured documents (e.g. dialogue records, mixed-format docs) but highest cost; generally only used in offline preprocessing pipelines.

Comparison of the three approaches

The most common pragmatic compromise: first use recursive character splitting for rough cuts, then use embedding similarity for semantic-boundary correction; only enable proposition chunking on core knowledge-base passages.

5. Query Parsing

User questions are often colloquial, vague, and contain multiple sub-intents — using them directly for retrieval, vector similarity will drift, missing genuinely relevant documents. The goal of “query parsing” is to use an LLM to make the question more retrieval-friendly before retrieval.

Core idea: before Retrieval, use one (or more) LLM calls to transform the Query, then retrieve.

Query Rewriting

Principle: directly use an LLM to rewrite the user’s original question into one (or more) new queries that are more precise and closer to the language of the knowledge-base documents.

Example: user asks “how to fix this bug” → rewritten: “how to fix an IndexError array-out-of-bounds exception?”

Why it works: Embedding models use symmetric similarity — the more alike the user’s words and the document’s words, the closer the vectors. Rewriting bridges the “colloquial ↔ document language” vocabulary gap.

1
2
3
4

# Rewrite prompt sketch
system = "You are a professional search assistant. Rewrite the user's question into a precise query suited for document retrieval. Output only the rewritten sentence."
rewritten = llm.chat(system, user_query)
results = vector_db.search(embed(rewritten), top_k=5)

• Pros: simple to implement; one LLM call; significantly improves vocabulary-mismatch
• Cons: the rewrite may drift from the original meaning; adds one LLM’s latency

Query Decomposition

Principle: faced with a complex query containing multiple sub-questions, let the LLM break it into several independent sub-questions; each sub-question retrieves separately; finally aggregate the answers.

Example: user asks “What are the pros and cons of Python vs JavaScript in web development, and which should I choose?”
→ sub-question 1: “What are the pros of Python for web development?”
→ sub-question 2: “What are the cons of Python for web development?”
→ sub-question 3: “What are the pros of JavaScript for web development?”
→ sub-question 4: “What are the cons of JavaScript for web development?”

Each sub-question retrieves separately; the per-sub-question retrieval results are merged and handed to the LLM for the final answer.

Why it works: a complex question often corresponds to multiple scattered pieces of information in the knowledge base. Without decomposition, it’s hard for a single query to hit all relevant passages at once; with decomposition each query is more focused, and retrieval precision improves dramatically.

• Pros: especially suited for multi-hop reasoning and comparison questions
• Cons: number of sub-questions is uncontrolled; multiple retrievals multiply cost; aggregation logic is complex

HyDE — Hypothetical Document Embeddings

Principle: instead of embedding the Query directly, first let the LLM generate a hypothetical “ideal answer”, then embed that hypothetical answer and use it for retrieval.

1

User Query → LLM generates "hypothetical answer" → Embed(hypothetical answer) → vector retrieval

Example: user asks “Why is HNSW fast to query?”
→ LLM generates a hypothetical answer: “HNSW is based on a hierarchical graph structure; at query time, it quickly locates a general area from the high-level sparse graph, then does fine-grained search layer by layer…”
→ Embedding this hypothetical answer → compared to embedding the question directly, the vector is closer to the real documents in the knowledge base

Intuition: question embeddings and answer embeddings are not in the same place in semantic space. A hypothetical answer is more like a “real document”; therefore its vector is closer to documents in the knowledge base, so retrieval Recall is higher.

• Pros: very effective for “knowledge-intensive Q&A”; doesn’t depend on keywords; purely semantically driven
• Cons: the hypothetical answer may contain hallucinations, but the retrieval stage doesn’t rely on the answer’s correctness, only its vector, so hallucinations don’t directly affect results
• Suited for: specialized-domain Q&A; scenarios where document phrasing differs greatly from question phrasing

Multi-Query

Principle: let an LLM generate N variants of the same question from multiple angles, retrieve each separately, then deduplicate and merge the results (often combined with RRF).

Example: original question “the limitations of RAG”
→ variant 1: “In what scenarios does RAG perform poorly?”
→ variant 2: “What are the drawbacks of Retrieval-Augmented Generation?”
→ variant 3: “Failure cases of RAG systems”

Each of the three queries retrieves separately; after merging, re-rank with RRF; final Top-K coverage far exceeds any single query.

• Pros: fills coverage blind spots of a single query; naturally pairs with RRF
• Cons: N embeddings + LLM call; latency and cost scale linearly

Summary / Comparison of Approaches

In engineering practice: the most common lightweight recipe is Query Rewriting + Multi-Query (N=3) + RRF — only one extra LLM call, and retrieval quality improves noticeably. HyDE and Decomposition are reserved for scenarios with extreme quality requirements (e.g. legal-doc Q&A).

6. Reranker & Re-ranking Strategy

Background: vector retrieval (the regular two-encoding kind called Bi-Encoder) is fast but coarse — it compresses the Query and Document into one vector each and takes the dot product, losing lots of fine-grained interaction information. A Reranker is the second-stage module that does fine-grained scoring and re-ranking on Top-N candidates after coarse retrieval.

Two-stage retrieval (Bi-Encoder) pipeline

1
2
3
4
5

User Query
    ↓
[Stage 1] Vector retrieval (Bi-Encoder + FAISS) → Top-100 candidates   ← fast but coarse
    ↓
[Stage 2] (Cross-Encoder) Reranker fine-grained scoring → Top-10 final results   ← slow but accurate

Cross-Encoder

Principle: concatenate the Query and Document and input them to the same Transformer, letting all tokens of the two text segments attend to each other — directly output a relevance score.

1
2

Input:  [CLS] Query tokens [SEP] Document tokens [SEP]
Output: a relevance score between 0 and 1

• Pros: full attention; most accurate scoring; the precision ceiling
• Cons: Document vectors can’t be precomputed — every query has to rerun each candidate document with the Query; 100 candidates = 100 model inferences; high latency
• Suited for: small candidate sets (within Top-100), scenarios that demand the highest precision

ColBERT (Contextualized Late Interaction over BERT)

As shown in the image, each word in the question will resonate with every word in the article, yielding more precise results.

Principle: Query and Document are still encoded separately (so Documents can be precomputed), but instead of compressing to a single vector, the vector of each token is preserved. The relevance score is computed via MaxSim: for each token of the Query (split by token), find the most similar token across all Document tokens, sum across.

1
2
3
4

Query    → [q₁, q₂, q₃, ...]     one vector per token
Document → [d₁, d₂, d₃, ...]    one vector per token (can be stored offline)

Score = Σ max_j sim(qᵢ, dⱼ)     ← "Late Interaction"

• Pros: Documents can be precomputed offline and stored; interaction is much richer than a single-vector Bi-Encoder
• Cons: large storage (one vector per token, longer docs cost more); slower than Bi-Encoder, faster than Cross-Encoder
• Position: the middle tier between precision and speed; suited for larger candidate sets (Top-1000)

Side-by-side comparison