---
title: "Prompt Grids for Multi-Model Structured Forecasting"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Prompt Grids for Multi-Model Structured Forecasting}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

# Purpose

Structured forecasting projects often cross several dimensions at once:

- many input rows,
- several prompt variants,
- several models or routes, and
- different numbers of repeated completions for different models.

Hand-written nested loops can run this design, but they also make it easy to
lose provenance, miscount calls, overwrite raw output, or accidentally give a
model more weight merely because it has more completions. This vignette shows
the neutral prompt-grid orchestration provided by `nalanda`.

All executed code in this vignette is local and no-cost. Live model calls are
shown with `eval = FALSE`.

`run_prompt_grid()` treats prompt variants as **independent alternatives**:
each variant starts a fresh conversation. This differs from
`simulate_treatment()`, where the elements of `prompt` are **ordered turns** in
one conversation. Use `simulate_treatment()` when conversational memory is part
of the treatment design; use a prompt grid when prompts are parallel variants
that must not influence one another.

# Inputs and configurations

Start with one row per forecast target. Any columns can be interpolated into a
prompt template.

```{r}
library(nalanda)

interventions <- tibble::tibble(
  condition_id = c("control", "cooperation", "perspective"),
  text = c(
    "A neutral informational message.",
    "A message emphasizing a shared cooperative goal.",
    "A message asking the reader to consider another person's perspective."
  )
)

prompts <- tibble::tibble(
  prompt_id = c("direct", "contextual"),
  prompt = c(
    "Forecast the effects of this intervention:\n{text}",
    paste(
      "Consider the likely study population and implementation context.",
      "Then forecast the effects of this intervention:\n{text}"
    )
  ),
  active = c(TRUE, TRUE)
)

models <- tibble::tibble(
  model_id = c("fast_a", "reasoning_a", "fast_b", "candidate_off"),
  model = c("model-a-fast", "model-a-reasoning", "model-b-fast", "model-c"),
  integration = c("route-a", "route-a", "route-b", "route-c"),
  family = c("developer-a", "developer-a", "developer-b", "developer-c"),
  temperature = c(0, 0, 0.3, 0),
  n_completions = c(2L, 1L, 3L, 5L),
  phase = c("first", "later", "later", "not_selected"),
  active = c(TRUE, TRUE, TRUE, FALSE)
)
```

`model_id` is a stable analysis label, while `model` and `integration` specify
the backend route. `family` is an analysis grouping; it need not be identical
to a provider name. Inactive rows remain in the configuration for provenance
but are not planned or run.

# Preview before spending

Use `plan_prompt_grid()` before constructing a response schema or
contacting a backend.

```{r}
smoke_plan <- plan_prompt_grid(
  data = interventions,
  prompt_variants = prompts,
  model_config = models,
  smoke_n = 1
)

smoke_plan[, c(
  "model_id", "family", "prompt_id", "n_rows",
  "n_completions", "estimated_calls"
)]
sum(smoke_plan$estimated_calls)
```

Here the smoke run uses one input row. The full plan is the same call with
`smoke_n = NULL`.

```{r}
full_plan <- plan_prompt_grid(
  data = interventions,
  prompt_variants = prompts,
  model_config = models
)

sum(full_plan$estimated_calls)
```

`estimated_calls` counts requested model responses: input rows multiplied by
model-specific completions. A backend may submit those responses concurrently
or in batches, but the response count is the useful quantity for budgeting.
Calling `run_prompt_grid(..., dry_run = TRUE)` returns this same plan.

# Define a structured forecast

The response type can contain any number of fields. For a 13-outcome benchmark,
one concise construction is:

```{r, eval = FALSE}
outcome_names <- sprintf("effect_%02d", 1:13)
response_type <- do.call(
  ellmer::type_object,
  stats::setNames(
    replicate(13, ellmer::type_number(), simplify = FALSE),
    outcome_names
  )
)
```

Use meaningful field names in a real study. The schema should describe the
scale and direction of each forecast clearly enough that every model returns
comparable numbers.

# Smoke, phase the spend, then resume

Cost-gated studies can begin with one inexpensive model. A smoke test remains a
deliberate fresh check on one input row and is kept separate from the full-data
task identity:

```{r, eval = FALSE}
first_model <- models |>
  dplyr::filter(model_id == "fast_a")

smoke <- run_prompt_grid(
  data = interventions,
  id_col = "condition_id",
  prompt_variants = prompts,
  response_type = response_type,
  model_config = first_model,
  smoke_n = 1,
  output_dir = "results/forecast-smoke",
  resume = TRUE,
  on_error = "continue"
)
```

After inspecting `smoke$results` and `smoke$errors`, run that inexpensive model
over the full inputs. Saving the returned list as one RDS file keeps results,
the call plan, task statuses, and errors together as a compact run bundle.

```{r, eval = FALSE}
phase_one <- run_prompt_grid(
  data = interventions,
  id_col = "condition_id",
  prompt_variants = prompts,
  response_type = response_type,
  model_config = first_model,
  output_dir = "results/forecast-checkpoints",
  resume = TRUE,
  on_error = "continue"
)
saveRDS(phase_one, "results/forecast-phase-one-bundle.rds")
```

When the first phase is accepted, activate the later models. Existing results
can be a results table, a previous run bundle, or an RDS/CSV path. The checkpoint
collector is useful when the shared directory also contains successful models
that are no longer active in the current configuration.

```{r, eval = FALSE}
prior <- collect_prompt_grid_results(
  "results/forecast-checkpoints",
  existing_results = "results/forecast-phase-one-bundle.rds"
)

later_models <- models |>
  dplyr::filter(active)

later_plan <- run_prompt_grid(
  data = interventions,
  id_col = "condition_id",
  prompt_variants = prompts,
  response_type = response_type,
  model_config = later_models,
  existing_results = prior,
  dry_run = TRUE
)

later_plan[, c(
  "model_id", "prompt_id", "configured_calls",
  "reused_calls", "pending_calls"
)]
sum(later_plan$pending_calls)

full <- run_prompt_grid(
  data = interventions,
  id_col = "condition_id",
  prompt_variants = prompts,
  response_type = response_type,
  model_config = later_models,
  existing_results = prior,
  output_dir = "results/forecast-checkpoints",
  resume = TRUE,
  on_error = "continue"
)

raw_forecasts <- full$results
full$tasks
full$errors
saveRDS(full, "results/forecast-full-bundle.rds")
```

Each successful model-prompt-completion unit is written to its own RDS file
before the workflow advances. The filename contains readable IDs and a hash of
the inputs and settings. With `resume = TRUE`, a compatible file is loaded
instead of rerun. Failed units are listed in `errors` and are deliberately not
checkpointed, so a later run retries them.

Every raw row also carries `task_hash` and `input_row`. The hash covers the
input table, prompt text, response specification, and effective model settings.
Consequently, the same readable IDs with revised prompt text or a revised
response schema remain pending rather than being silently reused. A complete
prior task must contain every expected `input_row` before it suppresses calls.

Older unhashed results are rejected by default. For a reviewed one-time
migration only, `trust_legacy_results = TRUE` verifies exact task IDs, inputs,
and stored prompt text before assigning current hashes. The caller must
independently verify the old model settings and response specification; save
the resulting hashed bundle and return to the default afterward.

The combined raw table retains the input metadata and adds `model_id`, `model`,
`family`, `prompt_id`, `prompt_template`, `completion`, `temperature`, `seed`,
`integration`, and `output_mode`. Keep both this combined table and the
per-completion files. Aggregation should never be the only saved artifact.

# Simulations or repeated expert forecasts?

The mechanics are identical, but the interpretation is not.

Repeated completions are **simulations** when the prompt and design treat each
draw as a synthetic respondent or stochastic realization from a target
population. The repetition count then describes simulated sample size, and
the prompt should define the simulated unit.

Repeated completions are **repeated expert forecasts** when the model is being
used as a forecaster and repeated sampling measures its within-model
variability. Those draws are not independent experts. Calling them separate
experts would overstate the number and diversity of information sources.

Temperature zero does not guarantee identical output across every hosted
backend, and a nonzero temperature does not by itself turn completions into
valid population simulations. That claim depends on the research design.

# Aggregate without accidental row-count weights

The raw mean is usually wrong when models have different completion counts or
belong to families with different numbers of models. The helper below returns
every stage rather than hiding a final scientific choice.

This small local table mimics one outcome from a completed run:

```{r}
mock_raw <- tibble::tibble(
  condition_id = "cooperation",
  family = c("developer-a", "developer-a", "developer-a", "developer-a", "developer-b"),
  model_id = c("fast_a", "fast_a", "fast_a", "reasoning_a", "fast_b"),
  prompt_id = c("direct", "direct", "contextual", "direct", "direct"),
  completion = c(1L, 2L, 1L, 1L, 1L),
  effect_support = c(0, 2, 3, 6, 10),
  effect_trust = c(2, 4, 5, 7, 9)
)

median_aggregated <- aggregate_model_forecasts(
  mock_raw,
  outcomes = c("effect_support", "effect_trust"),
  unit_by = "condition_id",
  family_col = "family",
  method = "median"
)

mean_aggregated <- aggregate_model_forecasts(
  mock_raw,
  outcomes = c("effect_support", "effect_trust"),
  unit_by = "condition_id",
  family_col = "family",
  method = "mean"
)

median_aggregated$prompt
median_aggregated$model
median_aggregated$family
median_aggregated$consensus
mean_aggregated$consensus
```

The hierarchy is explicit:

1. completions receive equal weight within a prompt,
2. prompt estimates receive equal weight within a model,
3. model estimates receive equal weight within a family, and
4. family estimates receive equal weight in the consensus.

Set `family_col = NULL` if the intended estimand gives every model equal weight
directly. The count columns (`n_completions`, `n_prompts`, `n_models`, and
`n_families`) make each stage auditable. A raw mean would instead give extra
weight to whichever model produced more rows.

`method = "mean"` is the backward-compatible default. `method = "median"`
uses medians at all four stages, which can make the primary estimate less
sensitive to an extreme completion while retaining the same equal-weight
hierarchy. Missing values are omitted separately for each outcome at each
stage; an all-missing group remains missing. Counts describe all configured
contributors, including contributors missing a particular outcome.

# Diagnose and compare forecasts without project-specific wrangling

`tidy_forecast_aggregation()` makes every stage available in one long table.
It works with one or many numeric outcomes, and does not assume names for a
study's conditions or outcomes.

```{r}
forecast_long <- tidy_forecast_aggregation(
  median_aggregated,
  unit_by = "condition_id"
)

model_forecasts <- forecast_long |>
  dplyr::filter(aggregation_level == "model")

# Disagreement among models for each condition and outcome
summarize_forecast_disagreement(
  model_forecasts,
  unit_by = c("condition_id", "outcome"),
  estimate_col = "estimate",
  contributor_col = "model_id"
)

# The existing agreement helpers consume this model-stage shape directly.
effect_models <- model_forecasts |>
  dplyr::filter(outcome == "effect_support")
model_pairwise_cor(
  effect_models,
  outcome = "estimate",
  unit_by = "condition_id",
  model_col = "model_id"
)

# Mean--median comparison at the consensus stage. A lookup is optional and
# lets a project normalize differences by its own outcome-scale widths.
widths <- tibble::tibble(
  condition_id = "cooperation",
  outcome = c("effect_support", "effect_trust"),
  scale_width = c(10, 10)
)
compare_forecast_aggregations(
  median_aggregated, mean_aggregated,
  stage = "consensus",
  unit_by = "condition_id",
  outcomes = c("effect_support", "effect_trust"),
  labels = c("median", "mean"),
  scale_width = widths
)
```

The comparison helper refuses to silently drop unmatched cells or resolve
duplicate identities. `summarize_forecast_disagreement()` is deliberately not
limited to models: use `contributor_col = "prompt_id"` and include `model_id`
in `unit_by` to describe prompt disagreement within a model and target.

# What remains downstream

`nalanda` handles configuration validation, expansion, pending-call budgeting,
strong-identity reuse, checkpoint collection, provenance, resume, and
mean/median hierarchical aggregation. The following choices stay
in downstream analysis because they depend on the study:

- which models and prompt variants belong in the estimand,
- whether a `family` groups developers, model lineages, or another dependence
  structure,
- calibration, transformations, bounds, and missing-data rules for the 13
  outcomes,
- uncertainty intervals and dependence-aware inference,
- robustness and sensitivity analyses across prompts, models, or families, and
- whether repeated completions can be interpreted as simulations at all.

That separation keeps the package API reusable without silently turning one
benchmark's scientific assumptions into defaults for every project.