B2B organizations face significant risks from fraudulent activities due to the high volume of transactions at lower price points. This paper addresses the challenges of modeling organizational behavior and minimizing false positives in fraud detection by aggregating individual account behavior to an organization (with multiple accounts) level. We propose a multi-stage framework leveraging various machine learning models for pre and post-transaction stages on an eCommerce website, supporting both real-time and batch inferences, and incorporating third-party data decision engines. Our experiments demonstrate 98% precision in pre-transaction and 96% precision in post-transaction stages, resulting in a 31.8% reduction in fraud actors with 98% precision, decreasing fraudulent Gross Merchandise Value by 77.7%, and cutting manual verification time by 95%.