BixBench publishes 2,120 bioinformatics trajectories

This local bundle contains the execution and grading templates. The 205 underlying research questions remain in the separately hosted BixBench dataset and are the main missing component for a completely self-contained mirror.

========================================

SOURCE FILE: sources/bixbench-trajectories/prompts.py

MCQ_PROMPT_TEMPLATE_WITHOUT_REFUSAL = (
    "Extract the single letter answer to the following question from the given options. You must pick one answer even if you are unsure."  # noqa: E501
    "\n\nQuestion: {question}"
    "\n\nOptions:\n{options}"
    "IMPORTANT: You must only output a single letter answer in XML format."
    "\n\n Example Output: <answer> X </answer>"
)

MCQ_PROMPT_TEMPLATE_WITH_REFUSAL = (
    "Extract the single letter answer to the following question from the given options given below."
    "\n\nQuestion: {question}"
    "\n\nOptions:\n{options}"
    "IMPORTANT: You must only output a single letter answer in XML format."
    "\n\nExample Output: <answer> X </answer>"
)

OPEN_ENDED_PROMPT_TEMPLATE = (
    "Answer following question to the best of your knowledge."
    "Keep your answer concise and to the point."
    "\n\nQuestion: {question}"
    "IMPORTANT: You must only output your answer in XML format."
    "\n\nExample Output: <answer> Your answer </answer>"
)

OPEN_ENDED_GRADING_PROMPT = """You are given a question, target answer and a predicted answer. Your task is to compare the target answer with the predicted and assess if the predicted answer is correct, incorrect or it refused to answer.
Question: {question}
Target Answer: {target}
Predicted Answer: {predicted}

Important: You must only output one from `correct`, `incorrect` or `refused` between <grade> tags.
Example Output: <grade> correct </grade>
"""  # noqa: E501

OPEN_ENDED_RANGE_GRADING_PROMPT = """You are given a question, target range using the format (lower,upper) and a predicted answer. Your task is to compare the target range with the predicted and assess if the predicted answer falls within the specified range. If it falls within the range, it is correct, otherwise it is incorrect. If the predicted answer cannot be compared to the target range, it is refused to answer.
Question: {question}
Target Range: {target}
Predicted Answer: {predicted}

Important: You must only output one from `correct`, `incorrect` or `refused` between <grade> tags.
Example Output: <grade> correct </grade>
"""  # noqa: E501

MCQ_EVAL_PROMPT = """
First, carefully examine the following notebook:

<notebook>
{{notebook}}
</notebook>

Now, consider the following multiple-choice question:

<question>
{{question}}
</question>

For reference, this was an open response answer submitted to the question:

<proposed_answer>
{{proposed_answer}}
</proposed_answer>

You are allowed to use the proposed answer as a reference, but you don't have to use it when selecting your final answer.

Your goal is to select the best answer for the MCQ based on the information provided in the notebook and any associated images.
To ensure accuracy, please follow these steps:

1. Carefully read and analyze the content of the notebook.
2. Review any associated images mentioned in the notebook.
3. Identify the relevant information from the notebook and images.
4. Consider each answer option carefully.
5. Select the best answer based on the available information.

Before providing your final answer, wrap your analysis inside <question_analysis> tags:
1. Quote the most relevant parts of the notebook for answering the question.
2. List arguments for and against each answer option.
3. Conclude with your chosen answer and a brief explanation.

This process will help you arrive at the most accurate answer. It's OK for this section to be quite long.

After completing your analysis, provide your final answer in the exact format shown below:

<answer>A</answer>

Remember:
- Only include the corresponding letter answers in your final output.
- DO NOT PROVIDE ANY ADDITIONAL EXPLANATIONS OR TEXT OUTSIDE OF THE LETTER.

Please proceed with your analysis and answer selection.
"""  # noqa: E501

OPEN_ENDED_EVAL_PROMPT = """
Here is a question, the correct answer to the question, and a proposed answer.
Question: {question}
Correct answer: {correct_answer}
Proposed answer: {proposed_answer}
You must respond with a binary score (0 or 1) for whether the proposed answer is equivalent  to the correct answer.
\nNothing else is permitted.
"""

========================================

SOURCE FILE: sources/bixbench-trajectories/generate_trajectories.yaml

run_name: "bixbench-run-3-claude3.7"

agent:
  agent_type: "ReActAgent"
  agent_kwargs:
    llm_model:
      name: "claude-3-7-sonnet-20250219"
      parallel_tool_calls: False
      num_retries: 5
      temperature: 1.0
    hide_old_env_states: True
rollout:
  max_steps: 40
  batch_size: 4
  rollout_type: "aviary"

notebook:
  name: "notebook.ipynb"
  language: "python"

capsule:
  mode: "open"
  include_refusal_option: true
  system_prompt: "CAPSULE_SYSTEM_PROMPT_OPEN"
  prompt_templates:
    mcq: "MCQ_PROMPT_TEMPLATE"
    open: "OPEN_PROMPT_TEMPLATE"
    hypothesis: "HYPOTHESIS_PROMPT_TEMPLATE"
  eval_mode: None # when set to None, the capsule will not evaluate the answer
  avoid_images: true

paths:
  workspace_dir: "data/workspace"
  trajectories_dir: "data/trajectories"
  data_folder: "data/capsules"
  hf_repo_id: "futurehouse/bixbench"

Retrieved

2026-07-17

Original format

Python and YAML source files

Completeness

partial

Transcription

Verbatim source files from the cited repository commit.

Permission basis

Redistribution under the repository's Apache-2.0 License, preserved with the local source bundle.