Vision transformers with Inductive Bias introduced through self-attention regularization

• Material Type: 博士論文
• Title: Vision transformers with Inductive Bias introduced through self-attention regularization
• Author/Editor: BARBOSA, MORMILLE LUIZ HENRIQUE
• Author Heading: BARBOSA, MORMILLE LUIZ HENRIQUE
• Publication Date: 2023-03-18
• Publication Date (W3CDTF): 2023-03-18
• Degree grantor/type: 創価大学
• Date Granted: 2023-03-18
• Date Granted (W3CDTF): 2023-03-18
• Dissertation Number: 甲第198号
• Degree Type: 博士（工学）
• Text Language Code: eng
• Subject Heading: Inductive Bias; Vision Transformer; Self-supervised Learning
• Target Audience: 一般
• Note (General): In recent years, the Transformer achieved remarkable results in computer vision related tasks, matching, or even surpassing those of convolutional neural networks (CNN). However, unlike CNNs, those vision transformers lack strong inductive biases and, to achieve state-of-the-art results, rely on large architectures and extensive pre-training on tens of millions of images. Introducing the appropriate inductive biases to vision transformers can lead to better convergence and generalization on settings with fewer training data. This work presents a novel way to introduce inductive biases to vision transformers: self-attention regularization. Two different methods of self-attention regularization were devised. Furthermore, this work proposes ARViT, a novel vision transformer architecture, where both self-attention regularization methods are deployed. The experimental results demonstrated that self-attention regularization leads to better convergence and generalization, especially on models pre-trained on mid-size datasets.
• Persistent ID (NDL): info:ndljp/pid/13023742; https://dl.ndl.go.jp/pid/13023742
• Collection: 障害者向け資料
• Collection (Materials For Handicapped People:1): テキストデータ
• Collection (particular): 国立国会図書館デジタルコレクション > デジタル化資料 > 博士論文; https://dl.ndl.go.jp/collections/A00014
• Acquisition Basis: 博士論文（自動収集）
• Date Accepted (W3CDTF): 2023-10-11T15:41:03+09:00
• Format (IMT): application/pdf
• Access Restrictions: 国立国会図書館内限定公開
• Service for the Digitized Contents Transmission Service: 図書館・個人送信対象外
• Availability of remote photoduplication service: 可
• Periodical Title (URI): http://hdl.handle.net/10911/00040960; https://soka.repo.nii.ac.jp/records/40967
• Data Provider (Database): 国立国会図書館 : 国立国会図書館デジタルコレクション; https://dl.ndl.go.jp