AssociationType 'Committee_Member_Contact' has a referential integrity constraint and cannot be mapped

I get the AssociationType 'Committee_Member_Contact' has a referential integrity constraint and cannot be displayed in the EDMX file. There is no relationship between PK and FK in my database. A table Committee_Membermay have several Contact's. I have createdreference constraints as 'Contact- it is primary and Committee_Memberdependent.

public partial class Committee_Member
{
    public byte[] Committee_Member_Id { get; set; }          
    public byte[] Contact_Id { get; set; }           
    public string Committee_Code { get; set; }
    public string External_Source_Name { get; set; }
    public Nullable<System.DateTime> External_Last_Update { get; set; }         

    public virtual Contact Contact { get; set; }
}
public partial class Contact
{
   public Contact()
   {
        this.Committee_Member_SPE = new HashSet<Committee_Member_SPE>();
   }
   public byte[] Contact_Id { get; set; }
   public string First_Name { get; set; }
   public string Last_Name { get; set; }
   ....
   public virtual ICollection<Committee_Member_SPE> Committee_Member_SPE { get; set; }
}

Any solution for this?