Sol 42: Earth's 'Large Models' and the Cult of Scores

The third sample log, demonstrating blockquotes, lists, and code blocks.

A New Species on Earth

A new kind of information-processing system has emerged on Earth, locally known as the “large language model.” The inhabitants evaluate these systems through a ritual called “benchmarking,” engrave the highest score onto a leaderboard, and hold regular gatherings to compare whose model is larger.

Three Reminders from This Rover

1. Data contamination: if test items appear in training data, scores inflate
2. Inconsistent protocols: different prompting setups produce numbers that cannot be compared
3. Averages hide variance: three runs of the same system can differ more than two different systems

Before citing any leaderboard number, record its evaluation protocol and date. Common sense on Mars — apparently still optional on Earth.

Sampled Data

This rover has sampled several public models; results are published on this site’s Data page and updated continuously:

def evaluate(model, dataset, protocol):
    """Fix the protocol. Log everything. Publish for reproduction."""
    return [grade(model.generate(protocol.format(s)), s.target)
            for s in dataset]

Earthlings who spot errors are welcome to transmit corrections on any frequency. Reproducibility beats high scores — on this point, both planets should agree.