- Introduced the `media_segment` tool to segment objects in video footage using SAM 2.1, allowing users to track objects and generate mask files.
- Implemented input validation for prompts, ensuring at least one point or a bounding box is provided for segmentation.
- Enhanced the output structure to include mask paths and tracking details, improving integration with the `<mask>` component in the editor.
- Added comprehensive documentation for the new tool, detailing usage, input parameters, and expected outputs.
- Updated related components and tests to support the new functionality, ensuring robust performance and user experience.
- Integrated the `ObjectMaskBar` into the `Toolbar`, allowing for dynamic display based on the selected tool.
- Updated the `Inspector` component to remove the deprecated `ObjectMaskPanel`, streamlining the interface.
- Refactored shortcut handling to include new actions for the object mask tool, improving user interaction.
- Enhanced object mask interaction logic to support dynamic operation modes (add/subtract) based on user input.
- Improved overall code organization and documentation for better clarity and maintainability.
- Added new `ObjectMaskPanel` component to manage object masks within the effects inspector.
- Introduced functionality for adding and managing object masks, including dropdown menus for selecting masks and controls for editing.
- Enhanced the `EffectsInspector` to support object mask interactions, including displaying existing masks and allowing users to track new objects.
- Implemented brush functionality for painting masks directly onto video clips, improving user experience for object segmentation.
- Updated cursor styles for the object mask tool to provide clearer visual feedback during interactions.
- Refactored related components and utilities to support the new object mask features, ensuring seamless integration with existing functionality.
- Updated `package.json` to include new model export for `./models`.
- Expanded `constants.ts` to define new types and structures for SAM 2.1 models, including `Sam2Model` and `Sam2Geometry`.
- Refactored `gpu.ts` to utilize dynamic image sizes and geometry for frame processing, improving flexibility in GPU operations.
- Enhanced `load.ts` to support model-specific loading and progress tracking, optimizing the user experience during model initialization.
- Updated `mask.ts` to redefine mask handling with logits and improved painting functionality for strokes.
- Refactored `tracker.ts` to integrate new model structures and enhance video tracking capabilities with corrected masks.
- Improved documentation and type definitions across the codebase for better clarity and usability.
- Introduced `readFloat16Initializer` function in `onnx.ts` to read float16 initializers from model bytes, improving model loading capabilities.
- Updated `createSession` function in `sessions.ts` to accept additional options for session configuration, allowing for enhanced flexibility during model execution.
- Modified the `load` function in `load.ts` to utilize the new initializer and pass position tokens to the `Sam2Video` constructor, optimizing video processing.
- Refactored `Sam2Video` class to handle position tokens, ensuring accurate representation of the vision encoder's positional embeddings during video tracking.
- Updated `export.py` to include model size as a parameter for ONNX graph exports, improving flexibility in model loading.
- Refactored the `load` function to accept a checkpoint string, allowing dynamic model loading based on specified size.
- Introduced new methods in `tracker.ts` for holding and previewing frames, enhancing the user experience during object segmentation.
- Updated type exports in `video.ts` to include `FrameRequest`, facilitating better integration with the tracking system.
- Improved documentation and comments throughout the codebase for clarity on new features and functionality.
- Deleted the `convert_fp16.py` script, which was responsible for converting ONNX models from fp32 to fp16.
- Added `export.py` script to export fixed-shape ONNX graphs for SAM 2.1 video tracking, ensuring compatibility with input parameters for image size and memory frames.
- Updated `constants.ts` to reflect changes in model repository links and fixed shapes for `IMAGE_SIZE` and `MEMORY_FRAMES`.
- Adjusted memory management and constants in various source files to align with the new export functionality.
- Introduced `convert_fp16.py` script for converting ONNX models from fp32 to fp16, optimizing memory usage while maintaining input/output compatibility.
- Updated `MODEL_REPO` in `constants.ts` to point to the new fp16 model repository, reflecting changes in model file sizes.
- Enhanced `weights.ts` to retain model files in the cache, improving file management during model loading.
- Introduced the `@diffusionstudio/sam2` package for video object segmentation and tracking in the browser using ONNX Runtime on WebGPU.
- Added essential files including `package.json`, `tsconfig.json`, and various source files for core functionality, constants, and GPU processing.
- Implemented model file fetching and caching from Hugging Face, along with memory management for tracking objects across frames.
- Included shaders for preprocessing video frames and transposing feature maps for memory attention.
- Established a comprehensive API for loading models, processing frames, and tracking objects with segmentation masks.
- Ensured type definitions and module exports for seamless integration with existing projects.