Procedure step recognition
By formalizing the task of procedure step recognition (PSR), along with an evaluation scheme, we encourage researchers to develop methods to automatically recognize the completion of steps, rather than the (partial) execution of activities. Additionally, PSR systems should explicitly leverage procedural knowledge and allow a flexible execution order for procedural tasks, when the procedure allows it.
Task definition
The objective of PSR is to extract an estimate of all procedure steps correctly performed by a person up to time t, based on sensory inputs X and a descriptive set of the procedural actions to be performed P. The predicted completed procedure steps at time t are defined such that $$ {\hat{y}_t = \mathcal{F} (X_t, \mathcal{P}). } $$
Crucially, this definition allows for real-time operation, since contrary to existing tasks, PSR does not require a full recording of the procedure as input.
Evaluation metrics
Procedure order similarity (POS). We propose to measure the quality of a predicted sequence order for an entire recording by comparing it with a similarity measure with respect to the ground-truth. This is approached as a string similarity problem since words consist of a sequence of characters, where order and type of character matters. We define POS as $$ {\textrm{POS} = 1 - \textrm{min}(\frac{\textrm{DamLev}(y, \hat{y})}{|y|},\hspace{0.1cm}1) ,} $$ where DamLev() is a weighted DamLev edit distance function.
F1-score.
Average delay. To complement the aforementioned metrics with a temporal component, quantifying the time between the ground-truth completion and corresponding recognition of a step.
IndustReal dataset
Construction-toy car
The IndustReal dataset contains 6 hours of recordings, demonstrating how 27 participants assembly and perform a maintenance task on a construction-toy car. IndustReal uses 3D printed parts and instructions for printing your own model can be found here.
Overview of the parts and components in IndustReal.
We are thankful to the STEMFIE project for open-sourcing their 3D printed toy construction sets.
Novel aspects of the dataset
Variety of execution errors. IndustReal features 38 errors, of which 14 are exclusive to the validation and test sets. Whilst some datasets already include procedural errors (e.g. omissions), IndustReal is the first to also include a variety of execution errors (e.g. wrong type of nut used).
Subgoal oriented execution. Other datasets are either “free-style” assemblies or contain a strict, step-by-step execution order. IndustReal combines these execution types with a subgoal-oriented assembly style, where participants are given flexibility to determine the execution order between subgoals. This approach more closely resembles industrial procedures, since it maintains a hierarchy in procedure execution whilst allowing for flexibility where possible. IndustReal contains 48 different execution orders.
Open-source geometries. Scalability is an important factor for many industrial tasks, where simultaneously the technical drawings are often available. Therefore, 3D models for all parts are published, to stimulate use of synthetic data in procedural action understanding, e.g. by sim2real domain adaptation or generalization.
3D printed parts. To ensure reproducibility, future availability of the model, and growth via community effort, all parts are 3D printed and open source.
Annotations
IndustReal is specifically created to stimulate research on the novel procedure step recognition task. However, to broaden the possible uses of the dataset, we provide action recognition and object detection (assembly state detection) labels alongside the PSR annotations:
Samples from a clip in the IndustReal dataset, demonstrating the modalities and annotations for all three tasks. Gaze is indicated by the cross, detected hand joints by the dots. AR: action recognition, ASD: assembly state detection, PSR: procedure step recognition.