9 Commits

Author SHA1 Message Date
konstantin-paulus
67e783e5c8 Add Media Segment Tool for Object Tracking in Video
- Introduced the `media_segment` tool to segment objects in video footage using SAM 2.1, allowing users to track objects and generate mask files.
- Implemented input validation for prompts, ensuring at least one point or a bounding box is provided for segmentation.
- Enhanced the output structure to include mask paths and tracking details, improving integration with the `<mask>` component in the editor.
- Added comprehensive documentation for the new tool, detailing usage, input parameters, and expected outputs.
- Updated related components and tests to support the new functionality, ensuring robust performance and user experience.
2026-09-28 15:35:40 +02:00
konstantin-paulus
3f38c5eb60 Enhance Object Mask Tool Functionality and UI Integration
- Integrated the `ObjectMaskBar` into the `Toolbar`, allowing for dynamic display based on the selected tool.
- Updated the `Inspector` component to remove the deprecated `ObjectMaskPanel`, streamlining the interface.
- Refactored shortcut handling to include new actions for the object mask tool, improving user interaction.
- Enhanced object mask interaction logic to support dynamic operation modes (add/subtract) based on user input.
- Improved overall code organization and documentation for better clarity and maintainability.
2026-09-28 12:27:37 +02:00
konstantin-paulus
b7866279fa Implement Object Mask Tool Enhancements and UI Integration
- Added new `ObjectMaskPanel` component to manage object masks within the effects inspector.
- Introduced functionality for adding and managing object masks, including dropdown menus for selecting masks and controls for editing.
- Enhanced the `EffectsInspector` to support object mask interactions, including displaying existing masks and allowing users to track new objects.
- Implemented brush functionality for painting masks directly onto video clips, improving user experience for object segmentation.
- Updated cursor styles for the object mask tool to provide clearer visual feedback during interactions.
- Refactored related components and utilities to support the new object mask features, ensuring seamless integration with existing functionality.
2026-09-28 09:41:49 +02:00
konstantin-paulus
88f74f8bfe Enhance SAM 2.1 model structure and GPU processing
- Updated `package.json` to include new model export for `./models`.
- Expanded `constants.ts` to define new types and structures for SAM 2.1 models, including `Sam2Model` and `Sam2Geometry`.
- Refactored `gpu.ts` to utilize dynamic image sizes and geometry for frame processing, improving flexibility in GPU operations.
- Enhanced `load.ts` to support model-specific loading and progress tracking, optimizing the user experience during model initialization.
- Updated `mask.ts` to redefine mask handling with logits and improved painting functionality for strokes.
- Refactored `tracker.ts` to integrate new model structures and enhance video tracking capabilities with corrected masks.
- Improved documentation and type definitions across the codebase for better clarity and usability.
2026-09-27 23:43:47 +02:00
konstantin-paulus
37f6195ba0 Enhance model loading and session creation for SAM 2.1
- Introduced `readFloat16Initializer` function in `onnx.ts` to read float16 initializers from model bytes, improving model loading capabilities.
- Updated `createSession` function in `sessions.ts` to accept additional options for session configuration, allowing for enhanced flexibility during model execution.
- Modified the `load` function in `load.ts` to utilize the new initializer and pass position tokens to the `Sam2Video` constructor, optimizing video processing.
- Refactored `Sam2Video` class to handle position tokens, ensuring accurate representation of the vision encoder's positional embeddings during video tracking.
2026-09-26 23:05:43 +02:00
konstantin-paulus
1882b539c9 Enhance SAM 2.1 video tracking functionality and update export parameters
- Updated `export.py` to include model size as a parameter for ONNX graph exports, improving flexibility in model loading.
- Refactored the `load` function to accept a checkpoint string, allowing dynamic model loading based on specified size.
- Introduced new methods in `tracker.ts` for holding and previewing frames, enhancing the user experience during object segmentation.
- Updated type exports in `video.ts` to include `FrameRequest`, facilitating better integration with the tracking system.
- Improved documentation and comments throughout the codebase for clarity on new features and functionality.
2026-09-26 16:45:06 +02:00
konstantin-paulus
70b2fa3ddf Remove convert_fp16.py script and introduce export.py for ONNX model export
- Deleted the `convert_fp16.py` script, which was responsible for converting ONNX models from fp32 to fp16.
- Added `export.py` script to export fixed-shape ONNX graphs for SAM 2.1 video tracking, ensuring compatibility with input parameters for image size and memory frames.
- Updated `constants.ts` to reflect changes in model repository links and fixed shapes for `IMAGE_SIZE` and `MEMORY_FRAMES`.
- Adjusted memory management and constants in various source files to align with the new export functionality.
2026-09-26 10:22:23 +02:00
konstantin-paulus
6d275f1a61 Add fp16 conversion script and update model repository links
- Introduced `convert_fp16.py` script for converting ONNX models from fp32 to fp16, optimizing memory usage while maintaining input/output compatibility.
- Updated `MODEL_REPO` in `constants.ts` to point to the new fp16 model repository, reflecting changes in model file sizes.
- Enhanced `weights.ts` to retain model files in the cache, improving file management during model loading.
2026-09-26 09:39:17 +02:00
konstantin-paulus
6c3901e543 Add SAM 2.1 video object segmentation and tracking package
- Introduced the `@diffusionstudio/sam2` package for video object segmentation and tracking in the browser using ONNX Runtime on WebGPU.
- Added essential files including `package.json`, `tsconfig.json`, and various source files for core functionality, constants, and GPU processing.
- Implemented model file fetching and caching from Hugging Face, along with memory management for tracking objects across frames.
- Included shaders for preprocessing video frames and transposing feature maps for memory attention.
- Established a comprehensive API for loading models, processing frames, and tracking objects with segmentation masks.
- Ensured type definitions and module exports for seamless integration with existing projects.
2026-09-25 23:36:58 +02:00