Pith. sign in

REVIEW 4 major objections 4 minor 47 references

AmbiK: Dataset of Ambiguous Tasks in Kitchen Environment

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces AmbiK, a dataset of 1,000 pairs of ambiguous and unambiguous kitchen instructions for robots, and shows that all five tested ambiguity-detection methods perform poorly on it.

desk verdict A useful, transparent new benchmark for ambiguity detection, but the evaluation metrics and the NoHelp ICR anomaly need to be fixed before the reported numbers can be trusted. read the letter →

arxiv 2506.04089 v1 pith:47X77RMF submitted 2025-06-04 cs.LG cs.AIcs.CLcs.RO

Anastasiia Ivanova , Eva Bakaeva , Zoya Volovikova , Alexey K. Kovalev , Aleksandr I. Panov This is my paper
classification cs.LGcs.AIcs.CLcs.RO
keywords ambiguousinstructionsembodiedagentstaskplanninglanguagemodelsclarificationrequestsconformalpredictionkitchenrobotbenchmarkdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces AmbiK, a fully textual benchmark for detecting ambiguous task instructions for robots in a kitchen environment. It contains 1,000 pairs of ambiguous and unambiguous instructions (2,000 tasks total), each with an environment description, a clarifying question and answer, user-intent keywords, and a task plan. The instructions are divided into three ambiguity types—human preferences, common-sense knowledge, and safety—based on whether a well-behaved robot should ask for help. The authors evaluate three conformal-prediction-based detectors and two baselines on GPT-3.5, GPT-4, Llama-2-7B, and Llama-3-8B, and report that all methods largely fail to distinguish ambiguous from unambiguous tasks. The central point is that existing methods and model logits are not reliable for this kind of ambiguity, and a shared benchmark is needed to make progress.

What carries the argument

The central object is the dataset itself: 1,000 paired tasks, with each pair sharing the same environment and a plan that differs only where ambiguity begins. The definition of ambiguity is operational: a step in the plan allows multiple possible choices, and a wrong choice may have undesirable consequences. The paired structure is the key mechanism, because it allows a direct comparison of how a method behaves on an ambiguous instruction and its unambiguous twin. Conformal prediction, a method that returns a set of candidate actions guaranteed with a user-specified probability to contain the correct one, is the technique underlying the three main detectors. The paper also introduces three new metrics—Correct Help Rate, Set Size Correctness, and Ambiguity Differentiation—alongside Intent Coverage Rate and Help Rate, to score both whether the robot asks for help and whether the content of its prediction set is right.

What would settle it

Running the same five methods with human raters instead of keyword matching—where raters judge whether each prediction set and each help decision is appropriate—would settle whether models really fail on AmbiK or whether the reported near-zero scores are artifacts of the automatic metrics.

Watch

Extended reading notes

Core claim

The paper's central claim is that AmbiK exposes a real gap: state-of-the-art ambiguity-detection methods, including conformal prediction approaches designed for embodied agents, perform poorly on ambiguous kitchen instructions across multiple LLMs. Under the paper's definition, an instruction is ambiguous when, given the environment, at least one plan step admits multiple choices and a wrong choice can lead to undesirable consequences. By pairing each ambiguous instruction with a minimally different unambiguous counterpart, the dataset makes it possible to measure whether a method's uncertainty actually tracks ambiguity. The reported results show that prediction sets rarely match the true set of correct options, methods rarely or always ask for help regardless of task type, and the logit-based uncertainty used by conformal prediction often fails to order options sensibly.

Load-bearing premise

The load-bearing premise is that the hand-written 'user intent' keyword lists and the rule that asking for help is always a mistake for safety and common-sense tasks correctly define what a good robot should do; if those are wrong, the reported scores do not measure ambiguity competence.

Editorial extensions

If this is right

  • If AmbiK becomes the standard benchmark, ambiguity detection methods can be compared on the same 2,000-text corpus instead of on different task sets.
  • A model that handles ambiguity correctly would show high Ambiguity Differentiation: larger prediction sets on ambiguous tasks than on their unambiguous counterparts; none of the tested methods reaches 10% on this metric except LofreeCP.
  • The near-zero Set Size Correctness means the prediction sets produced by conformal prediction are not aligned with the actual ambiguity, so improvements to CP calibration alone are unlikely to fix the problem.
  • The success of the simple Binary method over logit-based methods suggests that re-querying the model explicitly for certainty is more reliable than reading its token probabilities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's metrics would be stronger if validated against human judgments: Intent Coverage Rate and Set Size Correctness depend on hand-written keyword substrings, and Correct Help Rate assumes that asking for help on safety or common-sense tasks is always wrong, a policy that could be tested with human raters.
  • The zero-context design, where every cup in the scene makes 'put the cup' ambiguous, is a deliberate simplification; extending AmbiK with dialogue history would test whether models can use prior interaction to resolve references.
  • The same generation pipeline—object sampling, LLM task generation, ambiguity rewriting, human annotation—appears directly transferable to other rooms or domains, and a cross-domain version would reveal whether the failure is specific to kitchens or general.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces AmbiK, a fully textual English-language benchmark for detecting ambiguity in kitchen robot task instructions, consisting of 1000 pairs of ambiguous and unambiguous tasks (2000 tasks total) generated with LLM assistance and human-validated, with annotations that include environment descriptions, clarifying question-answer pairs, user intents, task plans, and a three-way ambiguity-type taxonomy (preferences, common-sense knowledge, safety). The authors also evaluate three CP-based methods (KnowNo, LAP, LofreeCP) and two baselines (Binary, No Help) on GPT-3.5, GPT-4, Llama-2-7B, and Llama-3-8B, reporting low values on their proposed metrics (ICR, SSC, AmbDif, CHR) and concluding that current methods perform poorly on AmbiK and that LLM logits are an inadequate approximation of uncertainty.

Significance. If the quantitative claims are supported, AmbiK would fill a real gap by serving as a unified textual benchmark for ambiguity detection in embodied agents, with a larger scale (1000 pairs) and richer annotation (clarification QA, plans, user intents) than KnowNo or existing textual datasets. The paper's strengths include a clearly described pipeline, human validation with reported agreement over 95%, public availability of the dataset and prompts, and a broad evaluation spanning five methods and four LLMs. The main contribution is the dataset itself, which appears well-motivated and potentially reusable. However, the benchmark's evaluation metrics are not validated, and the paper's central claim that current SOTA methods are strongly challenged depends directly on those metrics; that claim is currently weaker than the dataset contribution alone.

major comments (4)
  1. [Section 4.2, Appendix D, Table 8] The Intent Coverage Rate (ICR) is computed by exact substring matching against hand-crafted 'User intents' such as 'glass' (Appendix A). No validation is provided that these keyword lists cover the space of acceptable phrasings of correct actions, nor are they checked against human judgments of intent match. A clear symptom is that NoHelp, which always emits a single top-1 option, obtains ICR = 0.00 in every model and every ambiguity setting (Table 8). If a model outputs the correct action with different wording (e.g., 'clear container' vs 'glass'), ICR is 0 despite a correct behavior choice. This directly affects the main conclusion in Section 4.4 ('No Help method performs the worst') and the broader 'highly challenging dataset' narrative. The authors should validate the intent lists with human annotators, adopt a semantic/fuzzy matching approach, or provide a qualitative error analysis that rules out the possibility that low ICR values are measurement artifacts rather than genuine model failures.
  2. [Appendix D, Eq. (5), Figure 4] Set Size Correctness (SSC) is defined as the IoU between the prediction set and the 'Correct Set', which for PREFERENCES object ambiguity is the ambiguity shortlist from the dataset. This definition penalizes a model that outputs a superset containing the entire correct shortlist plus one additional object, assigning a low SSC even though the model has correctly identified the ambiguous objects and, under the paper's own CP decision rule (Section 4.1), would ask for help. Since the paper uses the low SSC results in Figure 4 to support the conclusion that methods are unable to align their CP sets with actual ambiguity, the metric should be supplemented with coverage-based measures (e.g., whether the prediction set contains the full shortlist) and/or evaluated with alternative scoring thresholds. Without such checks, the claim 'no method achieves even 20% of SSC' may overlook behaviors that are actually consistent with the intended help-asking policy.
  3. [Section 4.2, Eqs. (3)-(4), Table 9] The CHR metric assigns correctness based on a strong policy assumption: for SAFETY and COMMONSENSEKNOWLEDGE tasks, a method is considered correct only if it does NOT ask for help, while for PREFERENCES tasks it is correct only if it DOES ask for help. This policy is justified in Section 3.2 through Grice's maxims and is reflected in the annotation guidelines in Appendix I, but no empirical evidence is provided that human users actually expect a robot to refrain from clarifying for safety or common-sense ambiguities. The annotation guidelines themselves note that the boundary between safety and common-sense is 'conditional', and the CHR metric turns this asserted boundary into a binary scoring rule. The paper should either present human judgment data on expected help-asking behavior for each ambiguity type, clearly present CHR as one possible user-defined policy rather than ground truth, or refrain from using CHR as the sole basis for the observation in Section 4.4 that agents 'either never request help or always do so'.
  4. [Section 3.4 vs. Section 4.4] The dataset was generated with LLM assistance (unambiguous tasks via Mistral, ambiguous tasks and QA pairs via ChatGPT) and then evaluated on LLMs including GPT-3.5 and GPT-4. This is not a parameter-fitting circularity, since the evaluation does not train on the generated data, but it does introduce a distributional bias risk: the measured difficulty of AmbiK may partly reflect the generators' lexical and reasoning distributions rather than task ambiguity per se. The paper does not report any results split by generation model, nor does it compare against a human-written control subset. Reporting such a split or including a small independently authored evaluation set would materially strengthen the robustness of the conclusion that AmbiK is highly challenging for current methods.
minor comments (4)
  1. [General] There are numerous typos and naming inconsistencies, including 'Ambik' vs 'AmbiK' (e.g., Figure 1 caption), 'Ammbiguous task' in Appendix K, 'rompts' in the Appendix G heading, 'UNAMBIGUOUStasks' in the Figure 3 caption, and 'Sence' in Table 11. A careful copyedit is recommended.
  2. [Section 4.3, Tables 8 and 9] The paper states that experiments with the Flan-T5 model were conducted on half of the dataset, but it does not explain which half, why, or how the reader should interpret the rows labeled 'Llama-2-7B + FLAN-T5' when comparing with full-sample methods. This makes those results difficult to interpret and should be clarified.
  3. [Appendix C] The LofreeCP hyperparameters λ1 and λ2 are fixed to 0.1 for all experiments without a sensitivity analysis. A short robustness check over a few values would help confirm that the reported differences between methods are not an artifact of a particular hyperparameter choice.
  4. [Table 1] The table contains a stray '2' in the 'Number of household tasks' row (last column) that appears to be a formatting artifact; the table should be cleaned and each cell should contain only the intended values.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: evaluation is independent of dataset construction; metric validity concerns are correctness risks, not circularity.

full rationale

The derivation chain is not circular. AmbiK is a dataset and evaluation benchmark; the only derived claims are empirical evaluations of five methods on it. The CP-based methods are calibrated on a separate 100-example split (Section 4.3) and tested on 800 examples, so the reported set sizes and ICR values are not fitted to the test set. The user-intent keywords and ambiguity shortlists (Appendix A) are annotation artifacts, not quantities fitted from the tested models; a model output either contains them or not, independent of how the dataset was constructed. The paper's central challenge claim is read off ICR/SSC/HR/CHR tables, and while the hand-crafted keyword lists and CHR's asserted behavioral norm are validity risks (the paper itself says 'other metrics were proposed by us' in the Section 4.2 footnote, and no human-judgment validation of ICR/SSC is reported), invalid or unvalidated metrics are a correctness concern, not circularity. The KnowNo replication (Section 4.4, Table 7, E.5) provides an external anchor: the authors reproduce KnowNo's success and help rates on KnowNo data, so the evaluation pipeline is not wholly self-referential. Self-citations (e.g., Kovalev and Panov 2022; Kiseleva et al. 2022) appear only as related-work background and are not load-bearing. The use of ChatGPT-family LLMs to generate the data and then testing GPT models on it is self-referential in provenance only; if anything, it biases toward GPT success, and it does not make the evaluation result equivalent to the dataset's construction. Therefore no step in the paper reduces to its input by equation, fitting, or citation chain.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced. The main assumptions are definitional and behavioral. Two hand-set constants (LofreeCP lambdas) and several domain assumptions about human-robot communication carry the evaluation; the dataset itself is a constructed artifact, not a derivation.

free parameters (1)
  • LofreeCP hyperparameters lambda1, lambda2 = 0.1 each
    Fixed by hand across all experiments, not tuned on AmbiK; taken within the range from the original LofreeCP paper. Affects LofreeCP results.
assumptions (5)
  • domain assumption An instruction is ambiguous if at least one planning step allows multiple possible choices that could lead to undesirable consequences.
    Defines the target concept in Section 3.1; the entire dataset and metric design depend on this definition.
  • domain assumption Zero-context setting: ambiguity is evaluated without previous human-robot sessions.
    Stated in Section 3.1; makes preferences consistently ambiguous but diverges from real interactive settings.
  • domain assumption Optimal behavior for preferences is always to ask; for safety and common-sense tasks, never to ask.
    Encoded in CHR formulas (Section 4.2 and Appendix D). If safety ambiguity sometimes warrants clarification, CHR mislabels correct behavior.
  • domain assumption Grice's maxims govern human-robot instruction giving.
    Invoked in Section 3.2 to justify the taxonomy and expected behavior; a pragmatic assumption with no empirical support.
  • ad hoc to paper The three-category taxonomy (preferences, common-sense knowledge, safety) is exhaustive for kitchen instruction ambiguity.
    Section 3.2; other ambiguity types (spatial, linguistic) are deliberately folded in, but no argument establishes exhaustiveness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AmbiK: Dataset of Ambiguous Tasks in Kitchen Environment." pith.science (2026). https://pith.science/paper/47X77RMF

@misc{pith2026250604089,
  author       = {Pith},
  title        = {Pith review of: AmbiK: Dataset of Ambiguous Tasks in Kitchen Environment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/47X77RMF}},
  note         = {Machine review of arXiv:2506.04089}
}
read the original abstract

As a part of an embodied agent, Large Language Models (LLMs) are typically used for behavior planning given natural language instructions from the user. However, dealing with ambiguous instructions in real-world environments remains a challenge for LLMs. Various methods for task ambiguity detection have been proposed. However, it is difficult to compare them because they are tested on different datasets and there is no universal benchmark. For this reason, we propose AmbiK (Ambiguous Tasks in Kitchen Environment), the fully textual dataset of ambiguous instructions addressed to a robot in a kitchen environment. AmbiK was collected with the assistance of LLMs and is human-validated. It comprises 1000 pairs of ambiguous tasks and their unambiguous counterparts, categorized by ambiguity type (Human Preferences, Common Sense Knowledge, Safety), with environment descriptions, clarifying questions and answers, user intents, and task plans, for a total of 2000 tasks. We hope that AmbiK will enable researchers to perform a unified comparison of ambiguity detection methods. AmbiK is available at https://github.com/cog-model/AmbiK-dataset.

Figures

Figures reproduced from arXiv: 2506.04089 by the authors.

Figure 1
Figure 1. Ambiguity types in the Ambik dataset. challenging even for humans. However, such meth￾ods (Zhang and Choi, 2023; Chen and Mueller, 2023; Su et al., 2024; Testoni and Fernández, 2024) are often developed for question answering (QA) tasks and do not take into account important fea￾tures of embodied tasks. They differ from non￾embodied scenarios in their need for task speci￾ficity, grounding, and real-world interactivi… view at source ↗
Figure 2
Figure 2. Examples of ambiguous tasks in AmbiK across ambiguity types. For COMMON SENSE KNOWL￾EDGE, it can be unclear to the robot which kitchen item to use for toasting bread (a toaster). In SAFETY – which plate to use for buffalo wings (any microwave-safe one). Manipulator dataset (Ren et al., 2023). However, such classifications lack an internal system, as such semantic and linguistic divisions do not correlate with variou… view at source ↗
Figure 3
Figure 3. Intent Coverage Rate on AmbiK for UNAMBIGUOUS (a), PREFERENCES (b), COMMON SENSE KNOWL￾EDGE (c), and SAFETY (d) tasks. The NoHelp method has an ICR of 0 in all settings and is therefore not displayed. For the calibration stage of CP-based methods, 100 AmbiK examples were used, consisting of 50 unambiguous and 50 ambiguous examples, bal￾anced across different ambiguity types. Testing was conducted on 800 examples wit… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Set Size Correctness of CP-based methods. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 43 canonical work pages

  1. [4]

    IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022

    Iglu 2022: Interactive grounded language understanding in a collaborative environment at neurips 2022.Preprint, arXiv:2205.13771. Satwik Kottur, Seungwhan Moon, Alborz Geramifard, and Babak Damavandi. 2021. SIMMC 2.0: A task- oriented dialog dataset for immersive multimodal conversations. InProceedings of the 2021 Confer- ence on Empirical Methods in Natu...

    work page Pith review arXiv 2022

  2. [5]

    Introspective Planning: Aligning Robots' Uncertainty with Inherent Task Ambiguity

    Introspective planning: Guiding language- enabled agents to refine their own uncertainty.arXiv preprint arXiv:2402.06529. Justin Lidard, Hang Pham, Ariel Bachman, Bryan Boateng, and Anirudha Majumdar. 2024. Risk- calibrated human-robot interaction via set-valued in- tent prediction.Preprint, arXiv:2403.15959. Stephanie Lin, Jacob Hilton, and Owain Evans. ...

    work page Pith review arXiv 2024

  3. [6]

    DoRO: Disambiguation of referred object for embodied agents

    Doro: Disambiguation of referred object for embodied agents.Preprint, arXiv:2207.14205. 11 Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al. 2023. Robots that ask for help: Uncertainty alignment for large language model planners.arXiv preprint arXiv:2307.01928. Christ...

    work page Pith review arXiv 2023

  4. [7]

    heat”when both answers“heat

    API is enough: Conformal prediction for large language models without logit-access. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 979–995, Miami, Florida, USA. Association for Computational Linguistics. Alane Suhr, Claudia Yan, Charlotte Schluger, Stan- ley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. 2022....

    arXiv 2024

  5. [9]

    Pour honey into the glass food storage container until it is full

    work page

  6. [10]

    Plan for amb

    Close the honey jar or bottle. Plan for amb. task a detailed plan for the ambiguous task1. Locate the food storage container

    work page

  7. [11]

    Carefully open the honey jar or bottle

    work page

  8. [12]

    Pour honey into the food storage con- tainer until it is full

    work page

Show all 47 references
  1. [13]

    Experiments and results

    Close the honey jar or bottle. Start of ambiguity a number of plan point where ambiguity starts (Python-like indexing, 0 for the first point of the plan) 0 User intent keywords that should (not) be in the in- tented action (ground truth keywords) glass Variants possible action...

    2023

  2. [17]

    Add honey to vanilla yogurt

    Take a package of cheese from the fridge. You: A) pick up the greek yogurt cup from the fridge B) pick up cheddar cheese slices from the fridge C) pick up cottage cheese from the fridge D) pick up cream cheese from the fridge We: Apart from that, in the kitchen there is pa- pe...

  3. [20]

    Kitchen Robot, please use the vegetable peeler to peel the skin off the lemon in one continuous spiral, creating a lemon peel garnish for a cocktail or dessert

    Open the bottle of honey. You: A) use the bottle opener to open the bottle of honey B) use paper towels to open the bottle of honey C) open the bottle of honey without any tools D) use crackers to open the bottle of honey We: Apart from that, in the kitchen there is a bread kn...

  4. [21]

    Prepare a Cheesy Greek Yogurt Dip

    Take the lemon from the kitchen table. You: A) pick up the banana from the kitchen table B) pick up the lemon from the kitchen table C) pick up canned olives from the kitchen table D) pick up glass food storage container from the kitchen table _task_ We: Apart from that, in th...

  5. [25]

    Add honey to vanilla yogurt

    Take a package of cheese from the fridge. You: I will pick up cheddar cheese slices from the fridge. We: Apart from that, in the kitchen there is pa- per towels, a bottle opener, frozen mushroom pizza, frozen vegetable pizza, frozen pepperoni pizza, gin- ger biscuits, digestiv...

  6. [28]

    Kitchen Robot, please use the vegetable peeler to peel the skin off the lemon in one continuous spiral, creating a lemon peel garnish for a cocktail or dessert

    Open the bottle of honey. You: I will open the bottle of honey without any tools. We: Apart from that, in the kitchen there is a bread knife, a paring knife, a butter knife, a cut- ting board, a vegetable peeler, a potato masher, a plastic food storage container, a glass food ...

  7. [29]

    Prepare a Cheesy Greek Yogurt Dip

    Take the lemon from the kitchen table. You: I will pick up the lemon from the kitchen table. __task__ We: Apart from that, in the kitchen there is <DE- SCRIPTION>. We: You have received the task “<TASK>“ You created a plan to compete the task. <PREFIX> Your next action is: <AC...

  8. [30]

    Take a bowl from the kitchen table

  9. [31]

    Take a Greek yogurt cup from the fridge

  10. [32]

    Your next action is:

    Pour the Greek yogurt into the bowl. Your next action is:

  11. [33]

    Add honey to vanilla yogurt

    Take a package of cheese from the fridge. 19 You: I will pick up cheddar cheese slices from the fridge. Certain/Uncertain: Uncertain We: Apart from that, in the kitchen there is pa- per towels, a bottle opener, frozen mushroom pizza, frozen vegetable pizza, frozen pepperoni pi...

  12. [34]

    Take the bottle of honey from the kitchen table

  13. [35]

    Your next action is:

    Place the bottle of honey on the kitchen table. Your next action is:

  14. [36]

    Kitchen Robot, please use the vegetable peeler to peel the skin off the lemon in one continuous spiral, creating a lemon peel garnish for a cocktail or dessert

    Open the bottle of honey. You: I will open the bottle of honey without any tools. Certain/Uncertain: Certain We: Apart from that, in the kitchen there is a bread knife, a paring knife, a butter knife, a cut- ting board, a vegetable peeler, a potato masher, a plastic food stora...

  15. [37]

    You: I will pick up the lemon from the kitchen table

    Take the lemon from the kitchen table. You: I will pick up the lemon from the kitchen table. Certain/Uncertain: Certain __task__ We: Apart from that, in the kitchen there is <DE- SCRIPTION>. We: You have received the task “<TASK>“ You created a plan to compete the task. <PREFI...

  16. [38]

    • check for adequacy, correct if necessary If the example is completely strange (a recipe for mixing wine and mayonnaise), delete the line completely

    unambiguous_direct: This task (unambiguous and with a clear name of the objects) was generated using Mistral and previewed. • check for adequacy, correct if necessary If the example is completely strange (a recipe for mixing wine and mayonnaise), delete the line completely. • ...

  17. [39]

    • check for adequacy and compliance within the meaning of unambiguous_direct

    unambiguous_indirect: This task (unambiguous and with vague nam- ing of objects – paraphrasing, using demonstrative pronouns, etc.) was generated using ChatGPT. • check for adequacy and compliance within the meaning of unambiguous_direct. Convention- ally, a person should read...

  18. [40]

    do I wash vegetables?

    ambiguity_type, ambiguous_task: Ambiguous tasks of all three types and question- answer pairs were generated using ChatGPT. From the pref_raw, common_raw and safety_raw columns, you need to choose ONE of the most successful (logical and natural-sounding) ambiguous tasks. These...

  19. [41]

    question, answer: • select from the columns of the selected task type, check for adequacy, edit if necessary. The question should be logical, that is, before the question, an ambiguous task should be incompre- hensible to a person (in the case of preferences) or the work is no...

  20. [42]

    See Table 14 for examples

    amb_shortlist: Only for tasks of type PREFERENCES: a set of objects between which ambiguity is eliminated. See Table 14 for examples. Write and check that the set consists of at least 2 objects. Thank you for helping! J Appendix – Applying the data generating pipeline to other...

  21. [43]

    Object Categorization:Manually compile lists of objects grouped by semantic class (e.g., different types of chairs, electronic devices, etc.)

  22. [44]

    Unambiguous Task Generation:Sample en- vironmental objects and use the prompts pro- vided in Appendix F to prompt an LLM to generate unambiguous tasks specific to each environment

  23. [45]

    Ambiguous Task Generation:For each un- ambiguous task, generate ambiguous counter- parts using type-specific prompts

  24. [46]

    By following this structured approach, the pipeline can be seamlessly extended to new do- mains while maintaining consistency in task gener- ation

    Human Annotation:Conduct manual anno- tations following the guidelines in Appendix I to ensure data quality and consistency. By following this structured approach, the pipeline can be seamlessly extended to new do- mains while maintaining consistency in task gener- ation. K Ap...

  25. [47]

    Select a suitable knife from the knife block

  26. [48]

    Slice the lemon into thin slices

  27. [49]

    Slice the peach into thin slices

  28. [50]

    Slice the orange into thin slices

  29. [51]

    Store them in the food storage container in the fridge for later use

    Use the[glass]food storage container to store the sliced fruits in the fridge for later use KnowNo, chosen variants:[’C) use the glass food storage container to store the sliced fruits in the fridge’] LAP, chosen variants:[] (can’t answer) LofreeCP, chosen variants:[] (can’t a...

  30. [52]

    Fill[the tea kettle/the suitable appliance] with water from the sink

  31. [53]

    Place the tea kettle on the stove and turn on the heat to boil the water

  32. [54]

    While waiting for the water to boil, take out a black tea bag from the pantry

  33. [55]

    Once the water is boiling, pour it into a cup over the black tea bag

  34. [56]

    Let it steep for a few minutes before removing the tea bag

  35. [57]

    Serve hot black tea in a cup on the kitchen table for enjoyment. KnowNo, chosen variants:[’A) Fill the tea kettle with water from the sink.’] LAP, chosen variants:[] (can’t answer) LofreeCP, chosen variants:[’pick up the green tea bags from the kitchen table’, ’pick up the ene...

  36. [58]

    Unambiguous task:Kitchen Robot, please take out a clean sponge and wipe down the kitchen table

    Take out the[clean]sponge and wipe down the kitchen table. Unambiguous task:Kitchen Robot, please take out a clean sponge and wipe down the kitchen table. KnowNo, chosen variants:[’A) pick up the clean sponge from the kitchen’] LAP, chosen variants:[] (can’t answer) LofreeCP, ...

  37. [236]

    Guande He, Peng Cui, Jianfei Chen, Wenbo Hu, and Jun Zhu

    Springer. Guande He, Peng Cui, Jianfei Chen, Wenbo Hu, and Jun Zhu. 2023. Investigating uncertainty calibration of aligned language models under the multiple-choice setting.Preprint, arXiv:2310.11732. Zhiting Hu and Tianmin Shu. 2023. Language mod- els, agent models, and world...

    2023 arXiv

  38. [2022]

    arXiv preprint

    Scaling instruction-finetuned language models. arXiv preprint. Yinpei Dai, Run Peng, Sikai Li, and Joyce Chai. 2024. Think, act, and ask: Open-world interactive personal- ized robot navigation.Preprint, arXiv:2310.07968. Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowe...

    2024 arXiv

  39. [2024]

    Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, et al

    Self-play with execution feedback: Improving instruction-following capabilities of large language models.Preprint, arXiv:2406.13542. Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexa...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.