Document intelligence is a multimodal challenge, requiring both textual content and visual layout cues for accurate interpretation. Traditional text-based models often struggle to capture spatial and structural information, which is essential for document understanding tasks. While decoder-only language models (e.g. GPT-4, LLaMA) excel at text-based reasoning, they often overlook layout dependencies, limiting their effectiveness in tasks like Document Visual Question Answering (DocVQA) and Key Information Extraction (KIE), where text is arranged non-linearly. To address this, we introduce G-LLaMA, which integrates Gaussian biases into the attention mechanism of decoder-only models to enhance spatial reasoning. Unlike prior work that applied spatial biases to encoder-based architectures, our approach conditions LLaMA’s attention on spatial structure, improving layout awareness without requiring additional architectural modifications. Experimental results demonstrate that Gaussian biases significantly increase performance, yielding substantial improvements in both DocVQA and KIE tasks. These findings underscore the importance of explicit spatial conditioning in LLMs for understanding structured documents.