Large-scale trained models have shown significant success across various machine learning domains, leading researchers to explore their application in decision-making tasks. Hierarchical decomposition, particularly hierarchical reinforcement learning, is a vital approach to solving complex tasks by breaking them down into simpler sub-tasks. However, large-scale training a model under such a hierarchy remains challenging. Existing hierarchical reinforcement learning methods are formulated in online settings, which limits their scalability for large-scale training with sequence modeling. To address this limitation, we introduce a hierarchical structure into transformer-based offline RL. Our proposed approach, OF &D, is a contrastive learning framework that learns state-action temporal abstractions and hierarchical policies. We achieve state-of-the-art performance on the D4RL benchmark. Furthermore, this work paves the way for large-scale training in hierarchical reinforcement learning, facilitating the development of general long-horizon decision models.