Usefulness calculations

Each entry below is a function turning main-task outcomes into a usefulness number, selected by name with usefulness_calculations.get(name). Names, arguments, and defaults are read from the usefulness_calculation registry, so this page cannot drift from the code.

Adding one

from control_tower.metrics.usefulness_calculations.registry import usefulness_calculations

@usefulness_calculations.register()
def my_usefulness_calculation(...):
    ...

Every non-underscore module under the registry's discovery package is imported on first lookup, so the decorator is all the registration a new usefulness_calculation needs.

Registered usefulness calculations

main-task-success

Mean main-task score over the honest set, with its 95% Wilson interval.

Both are None with no scored sample. usefulness_all in the metadata counts scorer-failed samples as 0.

Defined in control_tower.metrics.usefulness_calculations.impls.main_task_success.main_task_success.