Personal Health Record (PHR) is beneficial to all stakeholders of the healthcare ecosystem. It is a patient controlled digital health record system. Medical history plays an important role in diagnosis and treatment for a person and stakeholders such surgeons, medical practitioners, pharmacist, laboratory technicians and nurse. They may need to access these records for their different needs during the entire treatment. To access these records, an effective access control mechanism is crucial. RBAC (Role based Access Control System) and ABAC (Attribute based Access Control system) are mostly used access control mechanisms. But because of prioritizing security of information system by healthcare institutions, patients have limited control on their healthcare records. Our study of PHR access control system is to identify most effective and secure access control mechanism of health record by all stakeholders. Though ABAC is an innovative solution for fine grained access control, its dependency on RBAC can lead to complexity and cost compliance. We have tried to address these issues by proposing a hybrid approach of integration of RBAC and ABAC with contextual access control. Our proposed framework also includes Ciphertext policy Attribute Based Encryption (CP-ABE) for protection of medical records. Trust factor plays a major role in the entire proposed model which is calculated based on attributes of documents and situational aspects. For well structured access control mechanism, rule based engine is integrated in the system. Our system is implemented fully in few controlled situations and proved effective in providing convenience in access with security.