Stakeholder engagement in comparative effectiveness research continues to gain national attention. While various methods are used to gather stakeholder expertise and form recommendations, evaluation of the stakeholder experience is often missing. The lack of evaluation prohibits assessing how effective and meaningful engagement practices are for enhancing research efforts and limits the ability to identify areas for future improvement. We propose that an evaluation plan of engagement processes be developed before stakeholder involvement begins and be required as part of a request for proposal or research grant where stakeholder input is being sought. Furthermore, we recommend the inclusion of six meta-criteria that represent normative goals of multiple studies: respect, trust, legitimacy, fairness, competence and accountability. To aid in the development of future evaluations, we have developed definitions for and matched specific examples of measuring each meta-criterion to serve a guide for others in the field.