Connor/cursebreaker-parser-rust
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2a3ba4ea4c40de6721fa8a838e96db6ad80267b5
cursebreaker-parser-rust/ROADMAP.md
Connor De Meyer 2a3ba4ea4c Phase 1 & 2
2025-12-30 12:16:52 +09:00

15 KiB
Raw Blame History

Unity Parser - Implementation Roadmap

Current Status

Repository Structure:

  • unity-parser/ - Main library crate (empty, needs implementation)
  • cursebreaker-parser/ - CLI binary for testing (empty, needs implementation)
  • unity-project-derive/ - Procedural macro crate (doesn't exist yet, needs creation)

Dependencies Already Configured:

  • Sparsey (not yet added to Cargo.toml)
  • serde, serde_yaml, serde_json
  • anyhow, glam
  • tokio (for async operations)
  • clap, rusqlite, sqlx (for CLI tool)

Phase 1: Core Foundation ✅ COMPLETE

Goal: Establish basic project structure and core types

Tasks

  • Create unity-parser/src/lib.rs with module structure

    • mod error; - Error types
    • mod types; - Core types (FileID, GUID, etc.)
    • mod meta; - .meta file parser
    • mod context; - ParseContext
    • mod asset; - AssetParser trait
    • mod world; - World builder

  • Add Sparsey ECS dependency to unity-parser/Cargo.toml

    sparsey = "0.11"

  • Create unity-project-derive/ procedural macro crate (deferred to Phase 7)

    • Initialize with cargo new --lib unity-project-derive
    • Add to workspace members in root Cargo.toml
    • Add proc-macro dependencies (syn, quote, proc-macro2)

  • Define core types in unity-parser/src/types.rs

    • FileID - Unity's local file identifier
    • GUID - Unity's global unique identifier
    • Reference - Represents {fileID: X, guid: Y} references
    • PropertyPath - Represents prefab override paths

  • Implement error types in unity-parser/src/error.rs

    • ParseError enum with variants:
      • InvalidYaml
      • MissingFile
      • InvalidReference
      • ComponentNotFound
    • Implement std::error::Error and Display

  • Create .meta file parser in unity-parser/src/meta.rs

    • Parse GUID from .meta files
    • Build GUID → file path mapping
    • Scan project directory for all .meta files

  • Implement ParseContext in unity-parser/src/context.rs

    • Store GUID mappings
    • Store FileID → Entity mappings
    • Cache loaded prefabs
    • Project root path

Completion Criteria: ✅ Can scan a Unity project and extract all GUIDs from .meta files.

See PHASE1_COMPLETE.md for detailed summary.

Phase 2: YAML Parsing 🔜

Goal: Parse Unity's YAML format efficiently with selective component loading

Tasks

  • Create unity-parser/src/yaml/mod.rs module

    • mod document; - Unity document structure
    • mod stream; - Streaming YAML parser
    • mod property_path; - Property path parser

  • Implement Unity document parser (yaml/document.rs)

    • Parse --- !u!XXX &fileID headers
    • Extract component type
    • Extract fileID anchor
    • Parse YAML body into generic structure

  • Create selective streaming parser (yaml/stream.rs)

    • Skip unknown component types without allocation
    • Only deserialize registered component types
    • Handle Unity's special YAML syntax quirks

  • Implement reference parser

    • Parse {fileID: X} (local references)
    • Parse {fileID: X, guid: Y} (external references)
    • Resolve references to Entity IDs

  • Implement property path parser (yaml/property_path.rs)

    • Parse paths like m_LocalPosition.x
    • Support array indices: m_Children.Array.data[0]
    • Apply overrides to parsed components

Completion Criteria: Can parse a simple .unity scene file and extract GameObject names.

Phase 3: Component System 🔜

Goal: Define component traits and implement built-in Unity components

Tasks

  • Create unity-parser/src/components/mod.rs

    • mod builtin; - Built-in Unity components
    • mod custom; - Custom component registry
    • mod traits; - Component traits

  • Define AssetParser trait (asset.rs)

    trait AssetParser {
    fn extensions() -> &'static [&'static str];
    fn parse(yaml: &YamlNode, context: &ParseContext) -> Result<Self>;
}

  • Implement GameObject pseudo-component

    • name: String
    • layer: u32
    • active: bool
    • tag: String

  • Implement Transform component (components/builtin/transform.rs)

    • local_position: Vec3
    • local_rotation: Quat
    • local_scale: Vec3
    • world_matrix: Mat4 (computed later)
    • parent: Option<Entity>
    • children: Vec<Entity>

  • Implement other common Unity components

    • MeshFilter (mesh reference)
    • MeshRenderer (materials)
    • Collider types (Box, Sphere, Capsule, Mesh)
    • Rigidbody
    • Camera
    • Light

  • Create component registry system

    • Map Unity type tags (!u!1, !u!4, etc.) to Rust types
    • Map MonoBehaviour script GUIDs to custom types
    • Provide lookup functions

Completion Criteria: Can parse a scene with GameObjects and Transforms into structs.

Phase 4: ECS Integration 🔜

Goal: Load parsed data into Sparsey ECS world

Tasks

  • Create unity-parser/src/world/mod.rs

    • mod builder; - World builder
    • mod entity_map; - FileID → Entity mapping

  • Implement WorldBuilder

    • Create Sparsey World
    • Track FileID → Entity mappings
    • Insert components into entities
    • Handle component dependencies

  • Create scene loading pipeline

    • Parse all GameObjects first (create entities)
    • Parse and attach components in second pass
    • Resolve all references
    • Return completed World

  • Implement basic scene loader

    pub fn load_scene(path: &Path, context: &ParseContext) -> Result<World>

  • Create query helper utilities

    • Wrapper around Sparsey queries
    • Type-safe component access
    • Optional ergonomic helpers

Completion Criteria: Can load a simple scene into Sparsey and query entities with specific components.

Phase 5: Prefab System 🔜

Goal: Support nested prefab instantiation with overrides

Tasks

  • Create unity-parser/src/prefab/mod.rs

    • mod instance; - Prefab instance handling
    • mod overrides; - Property override application
    • mod nesting; - Nested prefab support

  • Implement prefab loading

    • Load .prefab files like scenes
    • Cache loaded prefabs in ParseContext
    • Prevent circular references

  • Create PrefabInstance component parser

    • Extract source prefab GUID
    • Extract modification list
    • Parse property overrides

  • Implement prefab instantiation

    • Clone prefab entities into current world
    • Create new FileID mapping scope for each instance
    • Recursively handle nested prefabs
    • Maintain parent-child relationships

  • Apply property overrides

    • Parse property paths
    • Navigate to target component field
    • Apply override value
    • Support all field types (scalars, arrays, references)

  • Handle prefab variants

    • Load base prefab first
    • Apply variant overrides on top

Completion Criteria: Can load a scene with nested prefab instances and all overrides applied correctly.

Phase 6: Transform Hierarchy 🔜

Goal: Compute world-space transforms from local transforms

Tasks

  • Create unity-parser/src/transform/mod.rs

    • mod hierarchy; - Parent-child traversal
    • mod compute; - World matrix computation

  • Implement hierarchy builder

    • Build parent → children map
    • Detect root transforms (no parent)
    • Validate hierarchy (no cycles)

  • Implement world transform computation

    • Traverse hierarchy depth-first
    • Compute world matrix: parent.world * local
    • Handle scale, rotation, position correctly
    • Cache results in Transform components

  • Create post-process pass

    • Run after all entities and prefabs loaded
    • Single pass over all transforms
    • Update all Transform.world_matrix fields

Completion Criteria: World-space positions are correctly computed for nested GameObjects and prefab instances.

Phase 7: Procedural Macros 🔜

Goal: Implement ergonomic macro API for configuration

Tasks

  • Set up unity-project-derive/src/lib.rs

    • Add proc-macro crate type
    • Import syn, quote dependencies

  • Implement #[unity_parser(...)] configuration macro

    • Parse unity_types(...) list
    • Parse custom_types(...) list
    • Parse optional asset_types(...) list
    • Generate type registry
    • Generate parser configuration struct

  • Implement #[derive(Component)] macro

    • Generate field parsing code
    • Handle common field types automatically
    • Allow custom parsing attributes
    • Generate FromYaml trait impl

  • Create script GUID extraction tool

    • Scan project for .cs files
    • Parse file to find class name
    • Read corresponding .meta file for GUID
    • Build MonoBehaviour GUID → Rust type map

  • Generate type registration at compile time

    • Map Unity tags to built-in types
    • Map script GUIDs to custom types
    • Create static registry

Completion Criteria: User can declare desired types with single macro, no manual registration needed.

Phase 8: Caching Layer 🔜

Goal: Add optional SQLite caching for faster subsequent loads

Tasks

  • Create unity-parser/src/cache/mod.rs

    • mod schema; - Dynamic schema generation
    • mod storage; - SQLite operations
    • mod invalidation; - Cache validation

  • Implement schema generation

    • Create scenes table
    • Generate table per component type
    • Use reflection/macro data for columns
    • Handle relationships (foreign keys)

  • Implement cache storage

    • Serialize ECS world to SQLite
    • Store entity IDs and components
    • Store metadata (timestamps, hashes)

  • Implement cache loading

    • Deserialize from SQLite to World
    • Reconstruct entities and components
    • Restore references

  • Add cache invalidation

    • Hash scene and prefab files
    • Compare timestamps
    • Invalidate on source changes
    • User-controlled cache refresh

  • Add cache configuration

    • use_cache: bool parameter
    • Cache location configuration
    • Per-scene caching

Completion Criteria: Second load of same scene is 10x+ faster when cached.

Phase 9: CLI Tool 🔜

Goal: Create functional cursebreaker-parser binary

Tasks

  • Implement cursebreaker-parser/src/main.rs

    • Command-line argument parsing (clap)
    • Config file support (.env for game path)

  • Add CLI commands

    • parse <scene> - Parse and display scene info
    • export <scene> <format> - Export to JSON/SQL
    • list - List all scenes in project
    • cache clear - Clear cache

  • Implement progress reporting

    • Progress bars for large scenes
    • File count and size statistics
    • Error/warning summary

  • Add export formats

    • JSON (full scene dump)
    • SQL (INSERT statements)
    • CSV (per-component type)
    • Custom format (user-defined)

  • Configure Cursebreaker game path

    • Read from .env file
    • Example .env.example
    • Path validation

  • Create example configuration

    • Define Cursebreaker-specific components
    • Use #[unity_parser(...)] macro
    • Document component types

Completion Criteria: Can successfully parse Cursebreaker game and export data.

Phase 10: Testing & Documentation 🔜

Goal: Validate implementation and provide comprehensive documentation

Tasks

  • Create unit tests

    • Test YAML parsing
    • Test component deserialization
    • Test reference resolution
    • Test prefab instantiation
    • Test transform computation

  • Create integration tests

    • Test with minimal Unity project
    • Test with nested prefabs
    • Test with various component types
    • Test cache functionality

  • Test with Cursebreaker game

    • Load actual game scenes
    • Verify data correctness
    • Measure performance
    • Handle edge cases

  • Write API documentation

    • Document all public types
    • Document all public functions
    • Add usage examples
    • Document macro syntax

  • Update README.md

    • Quick start guide
    • Installation instructions
    • Basic usage examples
    • Feature list
    • License and contributing

  • Create examples

    • examples/basic_scene.rs - Load simple scene
    • examples/prefab_query.rs - Query prefabs
    • examples/export_json.rs - Export to JSON
    • examples/custom_component.rs - Define custom component

  • Performance benchmarks

    • Benchmark scene loading
    • Benchmark with/without cache
    • Benchmark selective parsing
    • Compare memory usage

Completion Criteria: All tests pass, documentation complete, README has working examples.

Future Enhancements (Post-MVP)

These are documented in DESIGN.md "Future Considerations" but not required for initial release:

  • ParserBuilder API for more flexible configuration
  • Support for added/removed components in prefab overrides
  • Component serialization versioning
  • More asset types (Materials, Textures, Animators, ScriptableObjects)
  • Binary cache format (faster than SQLite)
  • Helper query methods wrapping Sparsey
  • Parallel parsing (already in design, low priority)
  • Unity package support (Packages/...)

Key Design Decisions to Remember

  1. Sparsey over other ECS: Chosen for excellent insertion performance, lightweight. Query performance trade-off acceptable.
  2. Selective parsing: Memory efficiency by only parsing declared component types.
  3. Stream-based YAML: Skip unknown components without allocation.
  4. Post-process transforms: Compute world matrices after all entities loaded.
  5. Offline-only: No runtime integration, works on exported files only.
  6. Single macro: User declares all types in one place for convenience.
  7. Direct World exposure: Advanced users get full Sparsey access.

Dependencies Reference

unity-parser

  • sparsey - ECS backend
  • serde, serde_yaml, serde_json - Serialization
  • anyhow - Error handling
  • glam - Math types (Vec3, Quat, Mat4)
  • rayon - Parallel processing
  • walkdir - Directory traversal

unity-project-derive

  • syn - Parse Rust syntax
  • quote - Generate Rust code
  • proc-macro2 - Procedural macro utilities

cursebreaker-parser (CLI)

  • unity-parser - Core library
  • clap - CLI argument parsing
  • rusqlite/sqlx - SQLite access
  • tokio - Async runtime
  • indicatif - Progress bars
  • dotenv - .env file support

Getting Started (For Future Context)

To resume implementation:

  1. Check this roadmap to see current phase
  2. Read NOTES.md for any important decisions/gotchas
  3. Review DESIGN.md for architectural details
  4. Start with the first unchecked task in current phase
  5. Update checkboxes as you complete tasks
  6. Update NOTES.md with any new discoveries

Current Phase: Phase 2 (YAML Parsing) Next Action: Create unity-parser/src/yaml/mod.rs module structure

Reference in New Issue View Git Blame Copy Permalink