fmwork Tooling Enhancements: Monitoring, Benchmarking, and Analysis

[WarningRan]

Background

We need to enhance several utilities and features in our fmwork toolset to improve experiment capabilities, monitoring, and input options. These enhancements will make experiments more comprehensive, provide better insights, and support more diverse testing scenarios.

Overview of Required Enhancements

Monitoring Tools

• Memory Footprint: Add utilities to track memory usage during experiments
• Power Readouts: Implement power consumption monitoring

Experiment Workflow Improvements

• Benchmarking Loop: Add randomization options for sequences per repetition
• Input Options: Improve handling of synthetic and real datasets

Model-Specific Analytics

• MoE Model Statistics: Track expert triggering distribution for Mixture of Experts models

Plan