This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:

fact-checked

proofread

Large sequence models for sequential decision-making

Large sequence models for sequential decision-making
The difference between sequential decision-making tasks and prediction tasks, such as CV and NLP. (a) A sequential decision-making task is a cycle of agent, task, and world, connected by interactions. (b) In prediction tasks, tasks form a hierarchical structure. Credit: Muning Wen, Runji Lin, Hanjing Wang, Yaodong Yang, Ying Wen, Luo Mai, Jun Wang, Haifeng Zhang, Weinan Zhang

Transformer architectures have facilitated the development of large-scale and general-purpose sequence models for prediction tasks in natural language processing and computer vision, for example, GPT-3 and Swin Transformer.

Although originally designed for prediction problems, it is natural to inquire about their suitability in another important field, sequential decision-making and reinforcement learning problems, which are typically beset by long-standing issues involving sample efficiency, credit assignment, and partial observability, etc.

In recent years, sequence models, especially the Transformer, have attracted increasing interest in the RL communities, spawning numerous approaches with notable effectiveness and generalizability.

To inspire more investigation into this trending topic and empower more , e.g., robotics, automatic vehicles, and the automated industry, a research team led by Muning Wen published their survey in Frontiers of Computer Science.

Large sequence models for sequential decision-making
Detailed comparison between different Transformer-based methods for sequential decision-making. Credit: Muning Wen, Runji Lin, Hanjing Wang, Yaodong Yang, Ying Wen, Luo MAI, Jun Wang, Haifeng Zhang, Weinan Zhang.

The presents a comprehensive overview of recent works aimed at solving sequential decision-making tasks with sequence models such as the Transformer, by discussing the connection between sequential decision-making and sequence modeling, and categorizing them based on the way they utilize the Transformer.

These works suggest the potential for constructing a large decision for general purposes, that is, a large sequence model that can harness a vast number of parameters to perform hundreds or more sequential decision-making tasks, analogous to the way in which large sequence models have been leveraged for NLP and CV.

To examine the development of the Transformer in the field of sequential decision-making, the authors summarized recent works that convert the reinforcement learning problem into sequential form, to leverage sequence models for specific reinforcement learning settings.

Large sequence models for sequential decision-making
Analysis of what kind of data is used by these models (knowledge domain), how to understand the zero-shot generalization task (task indicator), what kind of component the sequence model is deployed as (what to pre-train), how to pre-train the model, and how to use the pre-trained model. Below is an explanation of the abbreviations in the table: Language model (LM), language and vision model (LVM), and behavior cloning (BC). Credit: Muning Wen, Runji Lin, Hanjing Wang, Yaodong Yang, Ying Wen, Luo MAI, Jun Wang, Haifeng Zhang, Weinan Zhang.

The authors' summary methods leverage diverse data to pre-train a large-scale sequence model for various downstream sequential decision-making tasks, inspired by the tremendous success of NLP and CV.

The team puts forth various potential avenues for future research to improve the effectiveness of large sequence models for sequential , encompassing , network architectures, algorithms, and efficient training systems. They hope this survey could inspire more investigation into this trending topic.

More information: Muning Wen et al, Large sequence models for sequential decision-making: a survey, Frontiers of Computer Science (2023). DOI: 10.1007/s11704-023-2689-5

Provided by Higher Education Press
Citation: Large sequence models for sequential decision-making (2023, December 15) retrieved 23 May 2024 from https://techxplore.com/news/2023-12-large-sequence-sequential-decision-making.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Explore further

A Hebbian memory that achieves human-like results on sequential processing tasks

2 shares

Feedback to editors