← Just Build It — the deck · evidence document

Eval — SAM3 Shoe-Parts Segmentation (VLM Pre-scan + Residual Loop)

JD item 6 (technical radar). All numbers below are read directly from output/sheet6-case/sam3-parts-side/segmentation_report.json (generated 2026-08-06T02:49:56.532Z), produced by src/agents/concept-3d/shoe-parts-pipeline.js, sam3-segment.js, residual-loop.js.

What we tested

Whether a VLM-generated per-shoe parts table + SAM3 (fal-ai/sam-3/image) prompted segmentation can reliably mask individual shoe components (laces, tongue, heel tab, midsole, etc.) from a single side-view image, well enough to drive the Region-Locked Batch Explorer's inpainting masks (src/agents/region-explorer/) — and what accuracy/cost that costs per run.

Setup

• Source image: output/sheet6-case/side.png, one side view, uploaded once and reused across all calls (uploadedImageUrl in the report) so only segmentation calls are billed, not uploads.
• Pipeline (runSidePartsPipeline, shoe-parts-pipeline.js:221): for each part in the Step-0 VLM-generated parts table, try part.prompt then fallback_prompts in order against SAM3 until a mask clears the confidence bar (min_instance_confidence: 0.5, config/shoe_pipeline_config.json:24) or prompts are exhausted; stop early once a good hit is found (tryPart, shoe-parts-pipeline.js:93-148).
• Residual loop (residual-loop.js): after all named parts run, find silhouette pixels not claimed by any part mask, connected-component them, VLM-name the largest ones (zone_type + material + tier), and register them as new parts without a further SAM3 call — this is a fallback for regions the prompt-based pass missed.
• QC threshold: a run needs human review once ≥2 required, universal-core parts fail (max_failed_core_parts_before_human_review: 2, shoe_pipeline_config.json:348).

Findings — per-part results (actual run)

* quarter_base came from the residual loop, not SAM3 — its score is a placeholder (score: 1 by convention in residual-loop.js:318), not a segmentation confidence.

Aggregate: 15 parts total, 12/15 (80%) status ok, 3/15 (20%) failed. Average score across the 12 SAM3-scored ok parts (excluding the residual entry) ≈ 0.79. Every failure scored exactly 0 — SAM3 returned zero masks for every prompt tried, not a low-confidence partial hit; this pipeline sees binary hit/miss more than graded degradation. Total: 22 SAM3 calls, $0.11 estimated cost (callCount * 0.005), across one side-view image.

failedRequired vs. actual failures — a scoping quirk found in the code, not the docs: the report's failedRequired list is ["collar", "upper_main"] — only 2 of the 3 failed parts. The third failure, air_window_side, is excluded because failedRequiredFinal only counts parts whose source is universal_core (or unset); air_window_side is sourced module_library despite being marked optional: false (shoe-parts-pipeline.js:326-331). Practical effect: this run's needsHumanReview: true was triggered correctly (2 ≥ threshold of 2), but a future run with one universal_core failure + two non-optional module_library failures would not hit the human-review gate under the current rule, even though 3 required parts failed.

Residual loop rationale, with real numbers: the residual pass ran 2 iterations. Iteration 0 found unclaimed-silhouette coverage of only 37.6% claimed by named parts, located 1 residual region above the 3%-of-silhouette size threshold, and the VLM named it quarter_base (leather, base_panel tier). Iteration 1 found 100.6% cumulative coverage (>100% because of mask dilation/overlap in the closing-kernel step) and 0 new regions, so the loop terminated. In other words: named-part prompts alone left 62% of the silhouette unclaimed after pass one — the residual loop, not prompt engineering, is what closed that gap.

Integrate now vs. wait

Risks

• Large structural panels are the weak point, not small hardware. The two base_panel/large parts that failed (upper_main, collar) are exactly the regions a colorway/texture batch explorer most wants to target broadly — meaning the current failure mode hits the highest-value use case hardest, not an edge case.
• Residual loop has no SAM3 validation of its own guess — it registers a mask purely from silhouette-minus-claimed pixel math plus a VLM label; the mask boundary can span multiple real parts if two adjacent regions were both missed by the prompt pass in the same connected component. score: 1 in the JSON should not be read as "confident," only as "auto-accepted."
• Single side-view only — this evaluation and the shipped pipeline (shoe-parts:side script) cover the side view exclusively; front/back/top part segmentation isn't evaluated here and isn't code-complete in this repo.
• Cost scales linearly with fallback depth — collar alone cost 3 calls for zero result; a part with a difficult prompt costs the same $0.005×N whether it succeeds or not, so a batch over 50 shoes with several persistently-hard parts could accumulate real cost without a corresponding accuracy gain unless fallback prompts are curated per failure.