Letter Train Project Technical Report

1. System Summary Overview

Letter Train is an educational app that teaches Sinhala letters by letting users trace ideal stroke paths and receive immediate feedback.

The project has three main parts:

Flutter application (mobile/desktop) that renders the UI, captures strokes, and scores user drawings.
Web demo (HTML/JS) that showcases the core tracing and rule‑based scoring in the browser.
ML training code (Python/PyTorch) that trains a sequence model exported to letter_scorer.tflite and used in‑app.

Letters උ, ම, ප, ග, ක

Scoring Rules + ML

Runtime Flutter, Web

2. Flutter App Architecture

Flutter

2.1 Entry and App Shell

lib/main.dart void main() => runApp(const LetterTrainApp());

The app starts with LetterTrainApp, a MaterialApp in lib/app/app.dart, which sets a green seed color and uses MainMenuPage as the home screen.

2.2 Navigation Flow

MainMenuPage: “Welcome” screen with cards for learning letters and two mini‑game placeholders.
HomePage: grid of Sinhala letters sourced from LetterSpec definitions.
LetterScreen: wrapper around the drawing canvas for a single letter, with a refresh button and navigation to practice mode and the next letter.

2.3 Letter Definition

lib/features/letters/models/letter_spec.dart class LetterSpec { final String title; final String svgData; final List<Offset>? startBase; final List<Offset>? endBase; }

Each letter (letterU, letterMa, letterPa, letterGa, letterKa) is defined by:

title: the Sinhala character.
svgData: a high‑resolution SVG path for the ideal stroke skeleton.
startBase / endBase: hint points in SVG coordinates indicating where to start and end tracing.

3. Tracing Canvas & Feedback

Core Logic

3.1 Canvas Component

The central widget is LetterPage (letter_canvas.dart), a stateful widget that:

Parses the letter’s SVG into a Flutter Path and fits it into a square drawing area.
Samples the path into hundreds of points to form a “spine” used for distance and direction checks.
Maintains a list of user strokes (List<List<Offset>>) captured from pointer events.

A lightweight LetterCanvasController lets parent widgets clear the canvas programmatically.

3.2 Start/End Guidance

On first stroke, the code checks whether the starting point lies close to the projected startBase polyline.
If the start is wrong, it displays “Start here” in red and refuses to begin the stroke.
At stroke end, it verifies proximity to both the logical path end and the endBase polyline; failing that, it vibrates and fully resets the canvas.

3.3 Visual Rendering

_SkeletonPainter: draws the letter skeleton in light grey, plus green “start here” and red “end here” guides with flags and arrows.
_DrawPainter: re‑draws user strokes segment‑by‑segment in green when close to the skeleton, red when off‑path.
A confetti system generates small falling colored rectangles when a letter is fully mastered.

4. Scoring & Modes

UX & Evaluation

4.1 Practice Modes

Guided mode: skeleton visible, stricter thresholds; success unlocks practice mode.
Practice mode: can hide the skeleton; success transitions to a “done” state with confetti and navigation to the next letter.

4.2 Geometric Scoring

The \_analyze method computes several rule‑based checks:

Size: bounding box must cover at least 25% of the canvas.
Direction: sample indices along the path should mostly increase as the user moves.
Order: stroke points must follow the intended progression along the path.
Accuracy: mean distance to the skeleton path must stay below a threshold.
Start/End: proximity to startGuide and endGuide polylines is required.

It also computes a base on‑path percentage by comparing segment length near the path to the total drawn length.

4.3 Haptics and Off‑Path Detection

Every move, the code measures distance from current pen position to the nearest sampled skeleton point.
Crossing a threshold triggers a heavy impact + vibrate and starts a periodic “buzz” timer whose frequency depends on how far off‑path the user is and how long they stay off.
Returning within the threshold stops the buzzing.

4.4 User Feedback Text

Based on both rules and ML scores, the app assembles readable feedback:

Positive: “Great job!”, “Now try without the guide”, “Yay! You completed <letter>.”
Corrective: combinations of “Make it bigger”, “Trace in the right direction”, “Use the right stroke order”, “Stay on the path”, “Start here”, “End here”, “Trace smoothly”.

5. ML Model & Data

Python / TFLite

The ML side lives under ml/ and trains a sequence model to score stroke quality.

5.1 Training Data

Dataset: LetterTraceDataset loads pre‑processed stroke records from datasets/letter_traces.
Each sample contains a time‑series of features plus target scores: overall quality, coverage, direction, order, scale, smoothness, per‑point probabilities, stroke breaks, and stylus dynamics.

5.2 Model Architecture

Input projection: linear projection + layer normalization from 8‑D features into a hidden size.
Encoder: bidirectional LSTM stacked under a TransformerEncoder.
Heads:
- Global head for six scalar scores.
- Per‑point head for point‑wise on‑path probabilities.
- Stroke‑break head for distribution over stroke bins.
- Dynamics head for stylus dynamics features.

5.3 Training Loop

Uses AdamW, OneCycleLR, gradient clipping, and optional CUDA AMP.
Multi‑term loss combining MSE, BCE, and BCEWithLogits across all heads with tuned weights.
Early stopping on validation loss; best model saved under checkpoints/advanced-letter-model.pt and later converted to TFLite.

5.4 In‑app Inference

At runtime, Flutter’s MLScorer:

Loads assets/models/letter_scorer.tflite via tflite_flutter.
Resamples the user’s variable‑length strokes into a fixed 256‑step sequence with engineered features (deltas, speed, distance to path, alignment, stroke breaks).
Runs the model and obtains scores for coverage, direction, order, scale, smoothness and per‑point alignment.
Blends model outputs with geometric rules to compute a final on‑path percentage and pass/fail decision.

6. Web Demo

HTML / JS

The web demo (index.html, app.js, styles.css) is a self‑contained browser implementation of tracing and geometric scoring.

HTML: sets up a single letter title, an SVG skeleton, a canvas overlay, a clear button, and feedback text.
CSS: provides a modern, mobile‑first layout with a square drawing board.
JavaScript:
- Hard‑codes one letter’s SVG path and fits it to a fixed 1000×1000 viewbox.
- Samples the path into points, collects pointer strokes, and draws them on canvas.
- Uses size, direction, order, and distance rules to print feedback like “Great job!” or combined hints.
- Registers a service worker for offline use as a small PWA.

7. Execution Flow

1. User opens app

Flutter boots, LetterTrainApp shows the main menu and letter list.

2. Letter selected

LetterScreen loads the letter’s SVG, fits the path, and initializes guides and samples; TFLite model loads in the background.

3. User draws

Pointer events build stroke arrays; drawings are colored green/red based on distance to the ideal path and haptics signal off‑path movement.

4. Scoring

Geometric checks and, if available, ML scores combine into an on‑path percentage and textual feedback; traces are stored to disk as JSON.

5. Progression

Passing guided mode unlocks practice; finishing practice triggers confetti and navigates to the next letter for continued learning.