Checkpoints and Resume#

Training lifecycle is controlled by Trainer options and checkpoint files under each model directory.

Checkpoint layout#

FF02_suffix/
├── best/
│   └── model_best.pth
└── last/
    └── model_last.pth

Committee models use model0.pth, model1.pth, … under best/ and last/.

Resume modes#

Trainer.resume (integer):

  • 0 — fresh training; may still load weights from pretrain_path if set

  • 1 — load last checkpoint weights, restart epoch counter behavior per trainer logic

  • 2 — full resume: optimizer, scheduler, early-stop state, epoch

Implementation: enerzyme/tasks/trainer.py.

Pretraining#

Modelhub.internal_FFs.FFxx.pretrain_path points to a previous run directory. Enerzyme resolves best/ or explicit paths via get_pretrain_path.

Typical in iterative AL:

pretrain_path: /task/FF02-SpookyNet-18
suffix: '19'

EMA#

  • use_ema: true

  • ema_decay: 0.999

  • ema_use_num_updates: true

Exponential moving average weights can stabilize late training. Check whether your evaluation uses EMA weights in the saved checkpoint.

Lightning multi-GPU#

Trainer.lightning: true enables PyTorch Lightning training (lightning_utils.py). Use for multi-GPU scaling; verify batch size and learning rate relative to single-GPU runs.

Logs and config snapshot#

Each enerzyme train invocation appends to logs/. The resolved YAML is written as config.yaml in the output directory — archive this file with checkpoints for reproducible predict/simulate.

Internal AL checkpoint#

Dataset active learning stores al_ckp.data when active_learning_params.resume: true. Distinct from model resume modes above.