Research Scientist

Research Scientist
Reports to: Director, Psychometrics and Research

Our Organization

We are a medical specialty certifying board serving anesthesiologists. Since 1938, we have been administering certification exams and today we take an innovative approach to certification and continuous learning. We foster practice standards that instill confidence and trust that board-certified anesthesiologists have the knowledge and skills to provide high-quality patient care. We are dedicated to elevating expertise in an evolving profession. Our mission is to advance the highest standards of the practice of anesthesiology. We work together with physician anesthesiologists to ensure they provide the best care possible for every patient, every day.

Position Description

The Research Scientist is a key member of the American Board of Anesthesiology (ABA)'s Assessment Services team, dedicated to advancing the ABA's research agenda and managing its research project portfolio. This role serves as a strategic and operational bridge between the Director, Psychometrics & Research, staff psychometricians, the ABA Research Committee, physician content experts, and other stakeholders. The Research Scientist is responsible for conceptualizing, designing, implementing, analyzing, and disseminating applied research studies that leverage the ABA's data to evaluate and strengthen the validity, fairness, effectiveness, and public value of ABA programs and anesthesiology practice. It provides focused leadership and execution capacity for a high-impact research agenda.

Education

• Ph.D. in educational research, measurement, quantitative or I/O psychology, statistics, biostatistics, health professions education, or a closely related quantitative field required.

Skills

• Advanced knowledge of research methodology (quantitative, qualitative, mixed methods), study design, and statistical analysis.
• Proficiency with one or more statistical programming languages (e.g., R, Python, SAS, or similar); familiarity with SQL and relational databases preferred.
• Strong scientific and technical writing skills; ability to communicate complex concepts to both technical and non-technical audiences.
• Excellent interpersonal skills and demonstrated ability to work effectively with physician leaders, volunteers, and multidisciplinary staff.
• Working knowledge of psychometric techniques (e.g., IRT, CTT, Multi-facet Rasch Model, reliability, validation studies) and ability to partner effectively with Senior Psychometricians.
• Strong project management, organization, and prioritization skills; ability to manage multiple concurrent projects and deadlines.
• Sound judgment, attention to detail, and commitment to data integrity, ethics, and confidentiality.
• Demonstrated commitment to service and continually exhibits a customer service culture.
• Demonstrated dependability for own actions and their consequences, and for the performance of the team functions and duties.
• Demonstrated flexibility in rapidly changing conditions.
• Independent self-starter with strong interpersonal and communication skills, both orally and in writing.
• Ability to work concurrently on multiple tasks and projects.
• Demonstrated willingness to keep technical skills and knowledge current.
• Ability to think logically and analytically in a problem-solving environment.
• Ability to work as well as part of a team and to manage relationships with vendors and consultants.
• Excellent interpersonal and leadership skills, with a strong ability to effectively interface and collaborate with varied constituent groups, including staff, Board of Directors members, volunteers, and external organizations.

Experience

• Minimum of 5 years of experience conducting applied research in testing, health professions education, outcomes research, or a related field.
• Demonstrated experience leading research projects from concept through analysis, interpretation, and dissemination.
• Experience working with large, complex datasets; experience in high-stakes testing, licensure/ certification, or medical education preferred.
• Record of peer-reviewed publications, conference presentations, or equivalent scholarly/technical output preferred.
• Experience working collaboratively with cross-functional teams (e.g., psychometricians, clinicians, data teams, academic partners).
• Proficiency with Microsoft Office products, particularly Word, Excel, and PowerPoint.
• Knowledge of or experience with the American Board of Medical Specialties (ABMS) a plus.
• Knowledge of or experience with the Accreditation Council for Graduate Medical Education (ACGME) a plus.

Specific Responsibilities

1. Research Leadership & Portfolio Management

• Collaborate with the Director, Psychometrics & Research and the Research Committee Chair to translate ABA strategic priorities into a focused, multi-year research agenda.
• Identify high-impact research questions using ABA examination, longitudinal, and outcomes data to support validity evidence, fairness/DEI analyses, outcomes research, and public value initiatives.
• Maintain an organized research portfolio and pipeline, including clear scopes of work, timelines, milestones, and status updates for active and proposed projects.

2. Study Design, Analysis & Execution

• Develop rigorous study designs, protocols, and analysis plans for quantitative, qualitative, or mixed-methods projects.
• Coordinate with Senior Psychometricians and data staff to implement appropriate psychometric and statistical methods; leverage their operational expertise without duplicating their core responsibilities.
• Ensure documentation of methodologies, decisions, and results that meet internal quality standards and external scientific expectations.

3. Stakeholder Engagement & Collaboration

• Serve as a key staff collaborator to the ABA Research Committee, supporting the Chair and members in advancing the ABA's research agenda.
• Engage physician content experts, volunteer leaders, external investigators, and partner organizations in the design and execution of research projects.
• Facilitate effective communication among internal teams to ensure alignment and resource coordination.

4. Dissemination & Communication

• Prepare and/or co-author manuscripts for peer-reviewed journals, conference abstracts/ presentations, white papers, technical reports, and internal briefing documents.
• Translate complex analytical findings into clear, actionable insights for ABA leadership, committees, and the Board of Directors.
• Contribute to communications that highlight ABA's research contributions and reinforce the credibility and impact of its certification programs.

5. Research Operations, Governance & Compliance

• Manage research workflows and timelines using sound project management practices and tools.
• Ensure adherence to ABA policies, data governance requirements, privacy and confidentiality expectations, and IRB/ethics standards where applicable.
• Promote reproducibility, transparency, and continuous improvement in all research activities.

Special Requirements

• Ability to travel on occasion up to 10% with overnight stays.
• Ability to lift up to 30 pounds maximum.
• Ability to sit for extended periods of time.
• Must be able to work onsite in Raleigh, NC at least two days per week.