🎨 react-native-mask-segment-canvas

A React Native 0.79 interactive mask segmentation library. The core export is the MaskSegmentCanvas component, consumable via npm package or npm link from any React Native project.

• 🧠 OpenCV (react-native-fast-opencv): mask semantic layout, baseboard patching, region extraction
• 🖌️ Skia RuntimeEffect (SkSL): single-pass full-screen shader blending original image + LAB low/high frequency texture color overlays
• ✂️ Skia Path: dashed outline highlights for regions
• 👆 Interaction: bottom color bar for brush selection (optional initialization) → tap a region to paint; tapping without a brush selected fires onPaintCallback with a hint; long-press without a brush previews the region's dashed outline

This repository serves as both the library source (src/index.ts) and a self-test demo (root App.tsx).

📌 For the recommended integration demo, see the example/ directory — it uses only the public API, fully simulating how a consumer project would integrate (including package.json, Metro configuration, and a complete reference App.tsx).

---

Table of Contents

• Overview
• Requirements
• Installation
  • Peer Dependencies
  • Postinstall Setup
  • iOS / Android Native Dependencies
  • Metro Configuration
  • Troubleshooting: Duplicate Module Errors
• Quick Start (Dev Demo)
• Basic Usage
  • Minimal Example
  • State Variables
  • Choosing Configuration Values
  • watchState & UI Guidance
• API Reference
  • Imports
  • Props: Image & Initialization
  • Props: Semantic Colors & Outline
  • Props: maskConfig
  • Props: pipelineConfig
  • Props: paintConfig
  • Props: interactionConfig
  • Props: UI Controls & Styling
  • Props: Callbacks
  • Ref Methods
  • Storage Convention
• Interaction Guide
• Integration Examples
  • Pre-warm PNG Cache (Recommended)
  • Passing Local Paths from an API
  • Draft Recovery
  • Custom Semantic Color Table
• Project Structure
• Dependencies
• Performance
  • Measured Reference (Dev Env + PNG Pre-warming)
  • Resolution vs pipelineConfig
  • interactive Estimation (Default Pipeline)
  • Device Tier (1080p, Default Pipeline)
  • Impact of Raising maxImageLongSide
  • Optimization Tips
• Notes
• Troubleshooting

---

🔭 Overview

MaskSegmentCanvas renders an original image with an overlaid semantic mask, allowing users to tap regions and apply colors. The pipeline:

1. 📥 Load the origin image and mask image (local file:// or remote http(s)://)
2. 🧩 Segment the mask via OpenCV into semantic regions (walls, ceiling, baseboard, etc.)
3. 🎨 Prepare LAB frequency-layer textures via SkSL for realistic color blending
4. 📐 Build Skia dashed-outline paths for each region
5. 👆 Interactive — users select a brush color and tap regions to paint; paint layers preserve the underlying texture
6. 💾 Save the composited result as PNG; export a JSON session for draft recovery

The component emits onWatch state transitions through the pipeline so the host app can show appropriate loading states.

---

📋 Requirements

• 🟢 Node.js >= 18 (recommended 20+)
• 🍎 Xcode 15+ (iOS)
• 🤖 Android Studio + JDK 17 (Android)
• 📦 CocoaPods (iOS)

---

📦 Installation

📦 Peer Dependencies

Install these in your host project (versions should match your host RN version):

npm install @shopify/react-native-skia react-native-reanimated react-native-fast-opencv react-native-fs buffer upng-js
# If using showDebugPickers (photo library picker)
npm install react-native-image-picker

🛠️ Postinstall Setup

This library relies on patch-package to patch react-native-fast-opencv. Your host package.json must include:

{
  "scripts": {
    "postinstall": "patch-package"
  },
  "devDependencies": {
    "patch-package": "^8.0.1"
  }
}

After installing this library, patches from node_modules/react-native-mask-segment-canvas/patches/ are applied automatically during the host's postinstall.

📱 iOS / Android Native Dependencies

cd ios && pod install && cd ..

Ensure the host project has completed Skia, Reanimated, and OpenCV native setup per each library's documentation.

🚇 Metro Configuration

When using npm link, a monorepo, or file: dependencies, add this library to watchFolders and use extraNodeModules + blockList to prevent duplicate module resolution:

const path = require('path');

module.exports = {
  watchFolders: [path.resolve(__dirname, '../MaskSegmentApp')],
  resolver: {
    nodeModulesPaths: [path.resolve(__dirname, 'node_modules')],
    extraNodeModules: {
      'react-native-reanimated': path.resolve(__dirname, 'node_modules/react-native-reanimated'),
      '@shopify/react-native-skia': path.resolve(__dirname, 'node_modules/@shopify/react-native-skia'),
      'react-native-gesture-handler': path.resolve(__dirname, 'node_modules/react-native-gesture-handler'),
      'react-native-fast-opencv': path.resolve(__dirname, 'node_modules/react-native-fast-opencv'),
      'react-native-safe-area-context': path.resolve(__dirname, 'node_modules/react-native-safe-area-context'),
      'react-native-fs': path.resolve(__dirname, 'node_modules/react-native-fs'),
    },
    blockList: [
      /\/MaskSegmentApp\/node_modules\/@shopify\/react-native-skia\//,
      /\/MaskSegmentApp\/node_modules\/react-native-reanimated\//,
      /\/MaskSegmentApp\/node_modules\/react-native-fast-opencv\//,
      /\/MaskSegmentApp\/node_modules\/react-native-gesture-handler\//,
      /\/MaskSegmentApp\/node_modules\/react-native-safe-area-context\//,
      /\/MaskSegmentApp\/node_modules\/react-native-fs\//,
    ],
  },
};

Strongly recommended — add this at the very top of the host index.js (before any business code):

import '@shopify/react-native-skia';

See example/metro.config.js and example/index.js for the complete configuration with all peer singleton packages.

⚠️ Troubleshooting: Duplicate Module Errors

Common symptoms:

• SkiaPictureView must be a function (received 'undefined')
• createAnimatedNode: Animated node[...] already exists

These are almost always caused by Metro resolving multiple copies of reanimated / skia / gesture-handler / fast-opencv / safe-area packages.

Best practice:

1. Copy the singletonPackages + extraNodeModules + blockList pattern from example/metro.config.js
2. At the top of your index.js, import gesture-handler → reanimated → skia in order
3. Restart Metro with --reset-cache and reinstall the app

See example/README.md for a detailed checklist and template.

---

🚀 Quick Start (Dev Demo)

The root App.tsx is a full self-test demo that imports directly from ./src.

cd MaskSegmentApp

npm install

cd ios && bundle exec pod install && cd ..

npm start

# In another terminal
npm run ios
# or
npm run android

To see how a consumer project integrates: go to the example/ directory and follow its README.md. It uses import from 'react-native-mask-segment-canvas' with standard package.json and Metro config, fully simulating a consumer environment.

---

💡 Basic Usage

🧑‍💻 Minimal Example

A complete, copy-pasteable example covering PNG pre-warming, state management, configuration, loading states, onWatch, and ref operations.

import React, { useEffect, useRef, useState } from 'react';
import { ActivityIndicator, Text, View } from 'react-native';
import MaskSegmentCanvas, {
  type MaskSegmentCanvasRef,
  type MaskSegmentSession,
  type MaskSegmentWatchState,
  type BgrColor,
  MASK_SEMANTIC_COLORS,
  prewarmPngBgrCacheAsync,
} from 'react-native-mask-segment-canvas';

/** Image paths prepared by the host app (local file:// or http(s)://) */
type ImagePaths = {
  origin: string;
  mask: string;
};

const INTERACTIVE_STATES: MaskSegmentWatchState[] = [
  'interactive',
  'mask_paths_ready',
];

export function PaintScreen() {
  const canvasRef = useRef<MaskSegmentCanvasRef>(null);

  const [imagePaths, setImagePaths] = useState<ImagePaths | null>(null);
  const [pathsError, setPathsError] = useState('');
  const [watchState, setWatchState] = useState<MaskSegmentWatchState | ''>('');
  const [errorMessage, setErrorMessage] = useState('');
  const [sessionDraft] = useState<MaskSegmentSession | null>(null);

  const isInteractive = INTERACTIVE_STATES.includes(
    watchState as MaskSegmentWatchState,
  );
  const isOutlineReady = watchState === 'mask_paths_ready';
  const isCanvasLoading =
    imagePaths != null &&
    watchState !== '' &&
    !INTERACTIVE_STATES.includes(watchState as MaskSegmentWatchState) &&
    watchState !== 'error';

  // Example: download images, then pre-warm PNG decode cache
  useEffect(() => {
    let cancelled = false;

    (async () => {
      try {
        const origin = 'file:///path/to/origin.png';
        const mask = 'file:///path/to/mask.png';
        await prewarmPngBgrCacheAsync([origin, mask]);
        if (!cancelled) {
          setImagePaths({ origin, mask });
        }
      } catch (e) {
        if (!cancelled) {
          setPathsError(e instanceof Error ? e.message : String(e));
        }
      }
    })();

    return () => {
      cancelled = true;
    };
  }, []);

  const handleSave = async () => {
    if (!isInteractive) return;
    const result = await canvasRef.current?.save({ destDir: undefined });
    console.log('saved', result?.filePath, result?.paintedCount);
  };

  if (pathsError) {
    return <Text>{pathsError}</Text>;
  }

  if (!imagePaths) {
    return (
      <View>
        <ActivityIndicator />
        <Text>Waiting for origin and mask images...</Text>
      </View>
    );
  }

  return (
    <View style={{ flex: 1 }}>
      {isCanvasLoading ? <Text>Loading: {watchState}</Text> : null}
      {watchState === 'interactive' ? (
        <Text>Paintable (outlines loading...)</Text>
      ) : null}
      {isOutlineReady ? <Text>Ready</Text> : null}
      {errorMessage ? <Text>{errorMessage}</Text> : null}

      <MaskSegmentCanvas
        ref={canvasRef}
        style={{ flex: 1 }}
        originUrl={imagePaths.origin}
        maskUrl={imagePaths.mask}
        semanticColors={MASK_SEMANTIC_COLORS}
        regionOutlineColor="rgba(20, 120, 235, 0.58)"
        maskConfig={{ blackThreshold: 30, maxRegionColors: 6 }}
        pipelineConfig={{ maxImageLongSide: 720 }}
        paintConfig={{ colorBaseOpacity: 0.88 }}
        interactionConfig={{
          initRegionFlashMs: 1000,
          enableInitRegionFlash: true,
        }}
        initialSession={sessionDraft ?? undefined}
        showDebugPickers={false}
        showToolbar={false}
        showColorBar
        showStatusRow={false}
        showOverlayButtons
        disabled={!isInteractive}
        onWatch={(state, durationMs, detail) => {
          setWatchState(state);
          console.log('[onWatch]', state, durationMs, detail);
        }}
        onPaintCallback={payload => {
          if (payload.kind === 'brush_required') {
            toast(payload.hint);
            return;
          }
          console.log('painted', payload.regionId, payload.regionName, payload.color, payload.configJson);
        }}
        onError={message => {
          setErrorMessage(message);
          setWatchState('error');
        }}
      />
    </View>
  );
}

📊 State Variables

State	Type	Purpose
imagePaths	`{ origin, mask }	null`
pathsError	string	Error message when path resolution or PNG pre-warming fails
watchState	`MaskSegmentWatchState	''`
isInteractive	derived	true when interactive or mask_paths_ready — operations are allowed
isOutlineReady	derived	true when mask_paths_ready — carousel dashed outlines are ready
isCanvasLoading	derived	Canvas init is blocking (not including PNG path waiting)
errorMessage	string	Segmentation/loading failure message written by onError
sessionDraft	`MaskSegmentSession	null`

⚙️ Choosing Configuration Values

Config	Use top-level prop when...	Use nested Config when...
Semantic colors	semanticColors={...} for most cases	maskConfig.semanticColors when paired with other mask params
Outline color	regionOutlineColor="..." for most cases	paintConfig.regionOverlayFill when also customizing the brush palette
Black threshold, max regions	—	maskConfig
Image processing size	—	pipelineConfig
Flash interval, tap tolerance	—	interactionConfig

Top-level props and nested Configs can coexist; top-level semanticColors / regionOutlineColor take priority.

🔄 watchState & UI Guidance

// Blocking loading (before regions + paint layers are ready)
const isLoading = ![
  'interactive',
  'mask_paths_ready',
  'error',
  '',
].includes(watchState);

// Allow tapping regions, selecting colors, painting (no need to wait for outline paths)
const canOperate =
  watchState === 'interactive' || watchState === 'mask_paths_ready';

// Carousel dashed outlines are fully ready (optional — can dismiss "outlines preparing" hint)
const isOutlineReady = watchState === 'mask_paths_ready';

// Show error screen
const hasError = watchState === 'error';

At interactive, detail.maskPathsReady is typically false; at mask_paths_ready, it is true. The gap is roughly ~100ms (async Skia path construction) and does not block tap-to-paint.

originUrl / maskUrl support:

• Local paths: file:///... or absolute paths
• Remote URLs: http(s)://... (the component handles download and decoding internally)

Legacy props originImgPath / maskImgPath are deprecated; use originUrl / maskUrl instead.

---

📖 API Reference

📥 Imports

import MaskSegmentCanvas, {
  type MaskSegmentCanvasRef,
  type MaskSegmentCanvasProps,
  type MaskSegmentSession,
  type MaskSegmentWatchState,
  type MaskSegmentWatchDetail,
  type BgrColor,
  type MaskSemanticColor,
  type PaintCallbackPayload,
  type PaintedRegionRecord,
  type PipelineConfig,
  type MaskSegmentConfig,
  type PaintConfig,
  type InteractionConfig,
  type SavePaintResult,
  MASK_SEMANTIC_COLORS,
  BASEBOARD_SEMANTIC_NAME,
  prewarmPngBgrCacheAsync,
  DEFAULT_PIPELINE_CONFIG,
  DEFAULT_MASK_CONFIG,
  DEFAULT_PAINT_CONFIG,
  DEFAULT_INTERACTION_CONFIG,
} from 'react-native-mask-segment-canvas';

Category	Names
Component	MaskSegmentCanvas (default)
Ref / Props types	MaskSegmentCanvasRef, MaskSegmentCanvasProps
Session / callback types	MaskSegmentSession, PaintCallbackPayload, PaintedRegionRecord, SavePaintResult
Watch types	MaskSegmentWatchState, MaskSegmentWatchDetail
Config types	PipelineConfig, MaskSegmentConfig, PaintConfig, InteractionConfig
Semantic colors	MASK_SEMANTIC_COLORS, BASEBOARD_SEMANTIC_NAME
Utilities	prewarmPngBgrCacheAsync
Runtime defaults	DEFAULT_*_CONFIG

🖼️ Props: Image & Initialization

Prop	Type	Required	Default	Description
originUrl	string	yes*	—	Origin image URL (file://, absolute path, or http(s)://)
maskUrl	string	yes*	—	Mask image URL (semantic color-block image; recommended same dimensions as origin)
originImgPath	string	—	—	Deprecated — use originUrl
maskImgPath	string	—	—	Deprecated — use maskUrl
initialSession	MaskSegmentSession	no	—	Draft restored from MMKV etc.; automatically calls loadSession after regions are ready
initialPaintColor	BgrColor	no	—	Optional. Initial custom brush color { b, g, r }; if omitted, no brush is selected by default; user must select a color or call ref.setPaintColor
initialPaintConfigJson	Record<string, unknown>	no	—	Optional. Accompanying brush config for initialPaintColor; passed back via onPaintCallback on successful paint

🎨 Props: Semantic Colors & Outline

Prop	Type	Default	Description
semanticColors	MaskSemanticColor[]	MASK_SEMANTIC_COLORS	Mask semantic recognition colors; equivalent to maskConfig.semanticColors
regionOutlineColor	string	rgba(20, 120, 235, 0.58)	Region dashed highlight color; equivalent to paintConfig.regionOverlayFill

Top-level props take priority over nested maskConfig / paintConfig.

MaskSemanticColor structure:

{
  name: string;   // Semantic name, e.g. wall / ceiling / baseboard
  hex: string;    // Display hex color
  bgr: { b: number; g: number; r: number }; // Must match mask pixel BGR channels
}

Built-in palette: MASK_SEMANTIC_COLORS (see src/utils/maskSemanticPalette.ts).

🧩 Props: maskConfig

Field	Type	Default	Description
semanticColors	MaskSemanticColor[]	built-in palette	Mask semantic colors (overridable by top-level semanticColors)
blackThreshold	number	30	Pixels with max(B,G,R) below this value are treated as black background
maxRegionColors	number	6	Maximum semantic regions retained
quantStep	number	64	Baseboard quantization step
baseboardMaxColorDist	number	42	Baseboard color distance threshold
baseboardStripQuantKeys	string[]	built-in keys	Baseboard strip quantization keys, format "b,g,r"
wallQuantKeys	string[]	built-in keys	Wall quantization keys
cabinetQuantKeys	string[]	built-in keys	Cabinet quantization keys
secondarySemanticNames	string[]	garageDoor, roof, eave	Secondary semantic names
secondaryMinPixelRatio	number	0.002	Minimum pixel ratio for secondary semantics
junctionHRadiusPx	number	24	Baseboard junction horizontal radius
junctionVRadiusPx	number	2	Baseboard junction vertical radius
kickBridgeHalfWPx	number	6	Baseboard horizontal gap bridge half-width
baseboardJunctionRowMarginPx	number	1	Baseboard junction row margin
baseboardJunctionVReachPx	number	2	Baseboard junction vertical reach
baseboardMinRunPx	number	2	Minimum run length for mask strips
splitWalls	boolean	false	Split wall mask into wall-1, wall-2… by texture boundaries
splitWallsMaxCount	number	8	Max wall sub-region count
splitWallsMinAreaRatio	number	0.002	Minimum area ratio for fragments (relative to total seg pixels)
splitWallsColorDistSq	number	1400	Connected-component chroma mean distance squared threshold
splitWallsChromaBlurRadius	number	5	Reserved: chroma smoothing radius
splitWallsNeutralChromaMax	number	14	White/gray wall low-chroma radius; forced boundary from colored walls

When splitWalls is enabled, the single wall region is replaced by multiple wall-N sub-regions, each independently paintable and undoable. Old sessions with regionName: 'wall' cannot map to new sub-region names and must be repainted.

🔬 Props: pipelineConfig

Field	Type	Default	Description
maxImageLongSide	number	720	Maximum long side for segmentation / pickMap / working area scaling
paintFreqMaxLongSide	number	480	Maximum long side for OpenCV LAB frequency layers
originPreviewMaxLongSide	number	360	Maximum long side for preview (main path uses working resolution)
maskPathMaxLongSide	number	480	Maximum long side for outline contour downsampling
minContourArea	number	100	Minimum contour area (scales proportionally with resolution)
contourApproxEpsilon	number	0.003	Contour polygon approximation coefficient
maxRegions	number	500	Maximum region count during segmentation

🖌️ Props: paintConfig

Field	Type	Default	Description
palette	BgrColor[]	6-color built-in	Bottom brush color strip
colorBaseOpacity	number	0.88	Base color opacity
lLightOpacity	number	0.50	L-channel overlay intensity
textureOpacity	number	0.85	High-frequency texture overlay intensity (stronger texture preservation)
lLowBlurKernel	number	7	Low-frequency Gaussian kernel (odd number)
lLowContrast	number	1.15	Low-frequency contrast
lLowBrightness	number	0.9	Low-frequency brightness
lHighGain	number	1.22	High-frequency gain
maskFeatherColor	number	1.6	Paint edge feathering (color) — soft-edge alpha radius, in pixels
maskFeatherTexture	number	0.9	Paint edge feathering (texture) — reserved/auxiliary
regionOverlayFill	string	rgba(20,120,235,0.58)	Dashed line / highlight fill color
regionOutlineStrokeWidth	number	4	Dashed outline stroke width

👆 Props: interactionConfig

Field	Type	Default	Description
pickMapSearchRadiusPx	number	14	Click pickMap search radius (pixels)
kickMaskPickRadiusPx	number	36	Baseboard mask pick radius
thinStripPadding	number	0.008	Thin strip (baseboard) tap expansion ratio
regionPadding	number	0.003	Normal region tap expansion ratio
initRegionFlashMs	number	1000	Duration each dashed outline stays during initial carousel (ms)
enableInitRegionFlash	boolean	true	Enable initial carousel animation

Full default constants: DEFAULT_MASK_CONFIG, DEFAULT_PIPELINE_CONFIG, DEFAULT_PAINT_CONFIG, DEFAULT_INTERACTION_CONFIG (exported from the package entry).

🎛️ Props: UI Controls & Styling

Prop	Type	Default	Description
showToolbar	boolean	true	Top toolbar ("Clear cache & re-segment")
showColorBar	boolean	true	Bottom brush color strip
showStatusRow	boolean	true	Segmentation/loading status text
showOverlayButtons	boolean	true	Bottom-left undo, bottom-right compare buttons
showDebugPickers	boolean	true	Photo library debug picker (set to false in production)
disabled	boolean	false	Disable paint interaction
style	ViewStyle	—	Outer container style
canvasStyle	ViewStyle	—	Canvas area style
undoButtonStyle / compareButtonStyle	ViewStyle	—	Overlay button styles
undoButtonTextStyle / compareButtonTextStyle	TextStyle	—	Overlay button text styles
undoButtonText	string	Undo (zh)	Undo button label
compareButtonText	string	Compare (zh)	Enter compare mode label
compareExitButtonText	string	Exit Compare (zh)	Exit compare mode label
renderUndoButton	(props) => ReactNode	—	Custom undo button renderer
renderCompareButton	(props) => ReactNode	—	Custom compare button renderer

📞 Props: Callbacks

Prop	Signature	Description
onWatch	(state, durationMs, detail?) => void	Initialization stage callback; durationMs is relative to this init start
onPaintCallback	(payload: PaintCallbackPayload) => void	Fires on successful paint, or when tapping a region without a brush selected
onError	(message, error?) => void	Segmentation or loading failure

PaintCallbackPayload (discriminated union, distinguished by payload.kind):

// Successful paint
{
  kind: 'painted';
  regionId: number;
  regionName: string;
  color: BgrColor;
  configJson?: Record<string, unknown>; // from setPaintColor / initialPaintConfigJson
}

// Tapped a valid region without selecting a brush (no paint performed)
{
  kind: 'brush_required';
  hint: string;       // e.g. "Please select a brush color first (bottom color bar or ref.setPaintColor)"
  regionId: number;
  regionName: string;
}

Example:

onPaintCallback={payload => {
  if (payload.kind === 'brush_required') {
    showToast(payload.hint);
    return;
  }
  savePaintRecord(payload.regionId, payload.color, payload.configJson);
}}

onWatch detail (MaskSegmentWatchDetail):

Field	Type	Description
regionCount	number	Current effective region count
maskPathsReady	boolean	Whether outline Skia paths are ready
freqLayersReady	boolean	Whether frequency Shader textures are ready
errorMessage	string	Failure description in error state

onWatch State Flow

init
  → images_loaded      Origin + mask read complete
  → mask_aligned       Mask dimensions aligned
  → mask_sampled       Mask pixel sampling complete
  → regions_ready      Region extraction succeeded
  → layers_ready       Paint texture layers ready (detail.maskPathsReady may still be false)
  → interactive        Interactive (can tap regions, select colors, paint)
  → mask_paths_ready   Outline paths ready (carousel dashed outlines can display; detail.maskPathsReady is true)
  → error              Failure (detail.errorMessage has description)

layers_ready / interactive may fire before outline paths finish computing. If the host dismisses a blocking loader at interactive, the user can already operate; carousel dashed outlines appear automatically after mask_paths_ready.

🔧 Ref Methods

Accessed via ref (type MaskSegmentCanvasRef):

Method	Signature	Description
reset	() => void	Undo last paint step (by paintHistory)
swap	(showOrigin?: boolean) => void	Toggle origin image comparison; omit arg to toggle, true/false to force
save	(options?) => Promise<SavePaintResult>	Composite and save PNG; options.destDir optional output directory
session	() => MaskSegmentSession	Export JSON-serializable session (for MMKV storage)
loadSession	(session) => void	Restore paint state (also available via initialSession)
setPaintColor	(color, configJson?) => void	Set current brush color; clears bottom color bar selection
setMaskConfig	(config) => void	Update mask config at runtime and re-segment
clearAllPaint	() => void	Clear all paint records
resegment	() => Promise<void>	Clear PNG cache and re-segment
getRegions	() => SegmentRegion[]	Snapshot of current region list
getPaintedRegions	() => PaintedRegionRecord[]	Snapshot of current paint records

SavePaintResult: { filePath, width, height, paintedCount, previewPath? }

Code examples:

const ref = useRef<MaskSegmentCanvasRef>(null);

ref.current?.reset();
ref.current?.swap();           // toggle
ref.current?.swap(true);       // force show origin

const result = await ref.current?.save({ destDir: '/path/to/dir' });

const session = ref.current?.session();
ref.current?.loadSession(session);

ref.current?.setPaintColor({ b: 100, g: 120, r: 140 }, { sku: 'paint-001' });
ref.current?.setMaskConfig({ semanticColors: customColors });

ref.current?.clearAllPaint();
await ref.current?.resegment();

const regions = ref.current?.getRegions();
const painted = ref.current?.getPaintedRegions();

save depends on the working buffer and pickMap being ready (typically after interactive); throws 'Image not ready, cannot save' if not ready.

💾 Storage Convention

Capability	Recommended Storage	Content
ref.save()	File system	Full-res PNG path
ref.session()	MMKV / AsyncStorage	JSON metadata (URLs, paint records, brush color, etc.)

MaskSegmentSession structure:

{
  version: 1;
  originUrl: string;
  maskUrl: string;
  painted: PaintedRegionRecord[];  // { regionId, regionName, color, configJson? }
  paintHistory: number[];
  currentColor?: BgrColor;
  currentColorConfigJson?: Record<string, unknown>;
  savedAt: number;
}

---

🎮 Interaction Guide

1. 🔁 Initial Carousel: After regions are ready, each region's dashed outline flashes sequentially per initRegionFlashMs (default 1s); stops on first user touch.
2. 🔍 Preview (no brush selected): Long-press a region to show dashed outline for the connected component under the touch point; tapping a black area shows no outline.
3. 🎨 Paint (brush selected): Tap a color in the bottom color bar or call ref.setPaintColor (or preselect via initialPaintColor), then tap a region to paint; tapping the same region again overwrites the color.
4. 💬 Tap without brush: No paint is performed; onPaintCallback fires with kind: 'brush_required', carrying a hint and target region info for the host to show a toast/modal prompting color selection.
5. ↩️ Undo: Bottom-left button or ref.reset(); steps backward through paint history one action at a time.
6. 👁️ Compare with Origin: Bottom-right button or ref.swap(); hides the paint layer to show the original image.

---

🧩 Integration Examples

🔥 Pre-warm PNG Cache (Recommended)

import { prewarmPngBgrCacheAsync } from 'react-native-mask-segment-canvas';

async function openPaintScreen(originUrl: string, maskUrl: string) {
  await prewarmPngBgrCacheAsync([originUrl, maskUrl]);
  navigation.navigate('Paint', { originUrl, maskUrl });
}

🌐 Passing Local Paths from an API

<MaskSegmentCanvas
  originUrl={localOriginPath}
  maskUrl={localMaskPath}
  showDebugPickers={false}
  showToolbar={false}
  semanticColors={MASK_SEMANTIC_COLORS}
  regionOutlineColor="#1e96ff"
  onWatch={(state, ms, detail) => {
    if (state === 'interactive') hideBlockingLoader();
    if (state === 'mask_paths_ready') hideOutlineHint();
  }}
/>

💾 Draft Recovery

const draft = JSON.parse(mmkv.getString('paint_draft'));

<MaskSegmentCanvas
  originUrl={draft.originUrl}
  maskUrl={draft.maskUrl}
  initialSession={draft}
/>

🎨 Custom Semantic Color Table

const gymColors: MaskSemanticColor[] = [
  { name: 'wall', hex: '#4363D8', bgr: { b: 216, g: 99, r: 67 } },
  { name: 'ceiling', hex: '#3CB44B', bgr: { b: 75, g: 180, r: 60 } },
  // ...
];

<MaskSegmentCanvas
  semanticColors={gymColors}
  maskConfig={{ blackThreshold: 30, maxRegionColors: 6 }}
/>

---

📁 Project Structure

MaskSegmentApp/                              # Repo root (npm package react-native-mask-segment-canvas)
├── App.tsx                                  # Dev self-test Demo (imports from ./src directly)
├── src/
│   ├── index.ts                             # Package entry (consumer: import 'react-native-mask-segment-canvas')
│   ├── components/
│   │   ├── MaskSegmentCanvas.tsx
│   │   └── MaskSegmentCanvas.types.ts
│   └── utils/
│       ├── maskSegmentation.ts
│       ├── maskSegmentRuntime.ts
│       ├── maskSemanticPalette.ts
│       └── ...
├── example/                                 # ★ Recommended: consumer-side integration demo
│   ├── App.tsx                              # Full example using only the public API
│   ├── index.js / app.json
│   ├── package.json                         # Required deps + "react-native-mask-segment-canvas": "file:.."
│   ├── metro.config.js / babel.config.js / tsconfig.json
│   └── README.md                            # How to integrate in a real project
├── patches/                                 # Shipped with the package; applied by host postinstall
├── ios/                                     # Root Demo native project (not published to npm)
└── android/

---

📚 Dependencies

Package	Purpose
@shopify/react-native-skia	Canvas rendering, Path, dashed strokes, Blend compositing
react-native-fast-opencv	Mask morphology, contour processing
react-native-fs	Layer caching, PNG save
react-native-image-picker	Demo photo library picker
react-native-reanimated	Skia animation dependency
react-native-safe-area-context	Safe area insets

---

⚡ Performance

The data below is based on the Demo test image (assets/test/origin.png 1080×1920, 6 semantic regions), default pipelineConfig, and onWatch durationMs (measured from init). These are empirical ranges, not strict benchmarks; actual device results vary with CPU, storage, and RN version.

📏 Measured Reference (Dev Env + PNG Pre-warming)

The Demo calls prewarmPngBgrCacheAsync([origin, mask]) before mounting the canvas, so PNG decoding hits the memory cache. Typical logs:

Stage	watchState	Approx. Duration	Notes
Mask aligned	mask_aligned	~160ms	Mask scaled to segmentation working resolution
Regions ready	regions_ready / mask_sampled	~320ms	Layout scan + baseboard + pickMap
Interactive	**interactive**	~320–450ms	Can tap regions, select colors, Shader paint
Outlines ready	mask_paths_ready	~430–550ms	~100ms after interactive; carousel outlines can display

interactive does not wait for outline paths; mask_paths_ready only affects the initial carousel and optional UI hints.

Same-image sub-step magnitudes (__DEV__ logs, default pipeline):

Sub-step	Approx. Duration	Working Resolution
OpenCV LAB high/low freq	~10–40ms	270×480
High/low freq Skia textures	~20–30ms	same
Layout scan + baseboard + pick table	~90–120ms	405×720 (1080p → longSide 720)
Full contour paths (async, non-blocking)	~80–150ms	270×480

📐 Resolution vs pipelineConfig

Compute-intensive steps are capped by maximum long side limits and do not scale linearly with 4K/8K origin images. Full PNG decoding still scales linearly with pixel count.

Step	Config Key	1080×1920 Actual Size	Scales with Origin Pixels
PNG decode	—	1080×1920 × 2 images	Yes
Mask seg / pickMap	maxImageLongSide: 720	~405×720	No (fixed when long side >720)
Shader high/low freq	paintFreqMaxLongSide: 480	~270×480	No
Working area Skia origin	same as maxImageLongSide	~405×720	No
Dashed outlines	maskPathMaxLongSide: 480	~270×480	No (does not block interactive)

⏱️ interactive Estimation (Default Pipeline)

Origin Spec	Relative to 1080p Pixels	With PNG Pre-warm	Cold Start (no pre-warm)
1080×1920	1×	320–450ms	450–700ms
1440×2560 (2K)	~1.8×	400–550ms	600–900ms
3840×2160 (4K)	~4×	500–750ms	800–1200ms
7680×4320 (8K)	~16×	0.8–1.5s	1.5–3s+

<300ms interactive: achievable on 1080p + pre-warm + default pipeline + high-end devices, but optimistic — do not treat as an all-device SLA.

📱 Device Tier (1080p, Default Pipeline)

Relative to the ~320ms dev-environment baseline:

Tier	Relative Multiplier	Pre-warm interactive	Cold Start
Flagship iOS / new flagship Android	0.8–1.2×	300–450ms	500–800ms
Mid-range Android	1.5–2.5×	500–800ms	700ms–1.2s
Low-end Android (4GB, old SoC)	2.5–4×	800ms–1.3s	1–2s+

Android overhead primarily comes from: JS ↔ OpenCV bridge, memory bandwidth/GC, Skia texture upload.

📈 Impact of Raising maxImageLongSide

Setting pipelineConfig.maxImageLongSide to 1280 (above the default 720) results in a segmentation working area of ~720×1280, roughly 3× the pixel count of the 720 tier:

Scenario	Default 720	Raised to 1280
1080p interactive (mid-range)	~320–800ms	500ms–1s+
Segmentation / pickMap duration	~90–120ms	~250–350ms

Higher precision for longer init time. To stay <500ms interactive, keep the default 720; reduce to 640 if needed.

💨 Optimization Tips

1. 🚀 PNG pre-warming (recommended): Call prewarmPngBgrCacheAsync after download/extraction and before navigating to the paint screen. Typically saves 100–250ms (greatest benefit on low-end devices).

import { prewarmPngBgrCacheAsync } from 'react-native-mask-segment-canvas';

await prewarmPngBgrCacheAsync([originPath, maskPath]);
// Then mount MaskSegmentCanvas

1. ⏱️ Loading timing: Dismiss the blocking loader at interactive; optionally listen for mask_paths_ready for "outlines preparing" hints.
2. 🖼️ Large images / low-end devices: Keep default maxImageLongSide: 720; optionally lower paintFreqMaxLongSide to 360.
3. 📷 4K assets: Downsample on the host side before passing in, or accept ~0.8–1.5s interactive (with pre-warm).
4. 🔍 Observability: Watch Metro logs for [MaskSegment], [⏱ ...] prefixes and onWatch durationMs.

---

📝 Notes

• The mask image should be a semantic color-block image with the same dimensions as the origin (black background + solid-color regions). Pixels with max(B,G,R) < blackThreshold (default 30) are excluded from segmentation.
• OpenCV segmentation runs on the JS thread; very large images may cause frame drops. Use pipelineConfig.maxImageLongSide to cap processing resolution.
• iOS photo library access requires photo permissions (only needed when showDebugPickers is enabled).
• semanticColors must match the semantic colors used in the backend/labeled mask; mismatch will cause recognition drift.

---

🔧 Troubleshooting

iOS pod install fails

cd ios
bundle install
bundle exec pod install --repo-update

Android build errors

cd android && ./gradlew clean && cd ..

Segmentation fails / zero regions

• Verify originUrl / maskUrl are accessible
• Confirm mask semantic colors match the semanticColors config
• Check Metro logs for [MaskSegment] / [⏱ ...] output

Dashed outlines misaligned / extra contours

• Outlines are generated from mask pixel external contours; long-press only shows the connected component at the touch point
• The initial carousel only shows the largest connected component for each semantic region