Authors : Alvin Chan, Yew-Soon Ong, Bill Pung, Aston Zhang, Jie Fu / Nanyang Tech, Amazon AI, Mila, Polytechnique Montreal

ICLR 2021 Poster Paper : https://openreview.net/pdf?id=VD_ozqvBy4W Code : -

Summary

• CoCon Block을 제안하여 전체를 tuning 하지 않고 해당 Block 만을 tuning 해 control 하는 방법 제안
• self-supervised learning approach로 다양한 loss로 더 나은 성능을 얻음
• 즉, GPT-2로 생성된 데이터로만 학습을 한다 (labeled data가 불필요)
• word/phrase level이 아닌 sequence level도 contents conditioning으로 사용 할 수 있게됨
• Generalization을 증가시킴
• CTRL, PPLM에 비해 성능이 좋음 (Human evaluation 도 좋다)

개인적 견해

• 앞으로는 PLM 전체를 학습시키기 보다는 일정 block을 학습시켜 control 하는게 대세가 될듯함
• 또는, prompt를 잘 구성하여 control 할 수 있을 것이라고 생각된다.
• Generative model의 controllability는 꾸준히 관심을 받으며 연구분야로 자리잡을 것이다.

Abstract

Pretrained Transformer-based language models (LMs) display remarkable natural language generation capabilities. With their immense potential, controlling text generation of such LMs is getting attention. While there are studies that seek to control high-level attributes (such as sentiment and topic) of generated text, there is still a lack of more precise control over its content at the word- and phrase-level. Here, we propose Content-Conditioner (CoCon) to control an LM’s output text with a content input, at a fine-grained level. In our self-supervised approach, the CoCon block learns to help the LM complete a partially-observed text sequence by conditioning with content inputs that are withheld from the LM. Through experiments, we show that CoCon can naturally incorporate target content into generated texts and control high-level text attributes in a zero-shot manner.