The association took the decision at its just concluded National Executive Committee meeting.
NBA said it took the decision so that the governor can have more time to focus on the killings in Southern part of Kaduna State.
Several Nigerian lawyers had in the past days registered their discontent over the decision of the leadership of the NBA to include Governor el-Rufai as a guest speaker in the conference.
The conference holds from August 26-29.
Other speakers at the conference include Vice President Yemi Osinbajo; Femi Gbajabiamila, Speaker of the House of Representatives; and Tanko Muhammad, Chief Justice of Nigeria; Olusegun Obasanjo, a former President; Ike Ekweremadu, former Deputy Senate President; and Abubakar Malami, Attorney-General of the Federation.