Fine-grained data authorization is essential for accomplishing the principle of least privilege in modern data governance. Existing approaches often rely on static policies or coarse-grained controls, which are inadequate for dynamic, task-specific scenarios. We propose a novel Task-Driven Fine-grained Authorization framework (TDFA) that automatically infers necessary data access permissions from natural language task descriptions. Our framework comprises two stages. First, the coarse-grained semantic filtering leverages embedding-based similarity to narrow down relevant data scope. Then, the fine-grained inference employs a prompt-engineered large language model (LLM) with reinforced fine-tuning. The resulting authorizations and their accompanying justifications ensure both interpretability and adaptability. We release a comprehensive benchmark dataset comprising schema information of 216 data tables and 973 annotated tasks including field-level authorizations and rationales. Experimental results show that our method achieves a new state-of-the-art performance.