SayComply: Grounding Field Robotic Tasks in Operational Compliance through Retrieval-Based Language Models

¹Stanford University, ²Field AI

Abstract

This paper addresses the problem of task planning for robots that must comply with operational manuals in real-world settings. Task planning under these constraints is essential for enabling autonomous robot operation in domains that require adherence to domain-specific knowledge. Current methods for generating robot goals and plans rely on common sense knowledge encoded in large language models. However, these models lack grounding of robot plans to domain-specific knowledge and are not easily transferable between multiple sites or customers with different compliance needs.

In this work, we present SayComply, which enables grounding robotic task planning with operational compliance using retrieval-based language models. We design a hierarchical database of operational, environment, and robot embodiment manuals and procedures to enable efficient retrieval of the relevant context under the limited context length of the LLMs. We then design a task planner using a tree-based retrieval augmented generation (RAG) technique to generate robot tasks that follow user instructions while simultaneously complying with the domain knowledge in the database.

We demonstrate the benefits of our approach through simulations and hardware experiments in real-world scenarios that require precise context retrieval across various types of context, outperforming the standard RAG method. Our approach bridges the gap in deploying robots that consistently adhere to operational protocols, offering a scalable and edge-deployable solution for ensuring compliance across varied and complex real-world environments.

Approach

At a high level, SayComply consists of the following steps. Prior to robot deployment, we build a hierarchical context database from various written manuals and instructions. Next, given a user query, relevant context source is retrieved using a tree-based RAG and LLM method. Finally, the compliant task planner generates robot tasks based on the retrieved context. Tasks are executed by the robot through behavior manager and the robot observations are stored in the database.

Operational Context Sources

Given that no existing benchmarks exist for field robotic task planning under operational compliance, we design a new set of experiments based on real-world use cases: 1) Industrial inspection in oil & gas and manufacturing, 2) Office operations and maintenance, and 3) Embodiment-aware operation in the field.

We compile all relevant instructions and manuals and simulate past observations for these three use cases, building a hierarchical database of context sources that includes 62 different manuals, instructions, and files.

SayComply Context Retrieval and Planning Example

The following video shows how SayComply retrieves relevant context sources and generates robot tasks for a given user query.

Experiments

We validate our approach through experiments in different real-world scenarios in industrial inspections that require grounding to operational compliance, alongside extensive experiments in NVIDIA’s high-fidelity Isaac simulator and on Boston Dynamics Spot robots. The video shows examples of hardware experiments in office operations and maintenance scenarios.