Edge-AI systems are increasingly being deployed in applications with stringent latency and security demands. However, existing access control solutions largely ignore the physical context of inference, leaving hardware vulnerable to contextual spoofing and physical attacks. This paper proposes a novel hardware-level access control mechanism that binds AI inference permission to real-time environmental fingerprints (EF) derived from ambient sensor data. An ESP32-based sensor node captures various environmental inputs, which are hashed using SHA-256 on an FPGA and compared against a trusted reference. Inference is only permitted when the environmental hash matches, enforcing context-aware execution via a gated TinyCNN accelerator. Experimental results demonstrate robust protection against unauthorized inference with minimal overhead, establishing a new physical root-of-trust for secure Edge-AI.