The agricultural sector remains a cornerstone to Africa’s economic transformation and the achievement of the Sustainable Development Goals (SDGs). However, the sector’s performance, in terms of productivity growth, has been sub-optimal when compared with performances in Asia, America, and Europe. Technical efficiency is a major contributor to the growth in agricultural productivity. However, there is limited research about the sources of technical inefficiency in agricultural production in Africa. This is critical given that Africa has problems with food security and high poverty. This study applied a parametric technique to estimate technical efficiency in Africa’s agricultural sector under heterogeneous technologies, while separating unobserved country-specific heterogeneities from transient and persistent technical efficiency. Data consisted of 49 African countries spanning from 1990 to 2016. Assuming a homogeneous technology, the estimated mean technical efficiency of 38.2%, suggests that, in Africa, about 62% of the potential agricultural output is untapped. However, considering heterogeneity in production technologies, the mean estimated technical efficiency value differed between the two classes. The numbers suggest that about 59% and 69% of untapped potential agricultural output in class one and class two groups, respectively. In all cases, persistent technical inefficiency hinders technical efficiency, an indication that government or regional agricultural policies should be long-term oriented. With the evidence of conditional convergence in technical efficiency, efforts to improve persistent technical efficiency can drive regional targets in food security and poverty alleviation and induce inclusive growth and sustainable development.