Best Practices in Medical AI Development
PDF Upload
Best Practices in Medical AI Development
WMA Educational Webinar Series on Artificial Intelligence in Medicine April 30, 2025
1. From Concept to Clinic: Understanding the AI
Development Journey
● Start with real clinical problems, not technology hype. Clinical problems are similar
worldwide, though workflows and administrative processes differ between institutions.
Use design thinking: interview and observe stakeholders to thoroughly understand the
real problem before developing solutions.
● Validate the problem before seeking technological solutions. Example: What appeared
to be a hospitalist-physician communication issue was discovered to be specifically
about discharge processes. Creating follow-up appointments from the hospital to
outpatient services led to 93% patient attendance and 12% reduction in readmissions.
● Consider the appropriate technology approach. Early AI applications focused on
imaging as data was already digital. Before generative AI (pre-2023), machine learning
required thousands of data points to prove a single algorithm. Generative AI has
transformed healthcare technology implementation, requiring less data and offering
more flexibility.
● Build minimum viable products by adapting existing technologies when possible.
Example: Defense industry image analysis technology designed to detect changes in
maps was successfully adapted for mammogram analysis to identify year-over-year
changes, flagging differences for physician review rather than attempting specific
diagnoses.
● Test thoroughly before implementation. Always obtain IRB approval and conduct
comparative studies with and without the technology. Successful integration depends
on embedding tools directly into clinical workflow, such as AI algorithms for brain
hemorrhage detection that prioritize abnormal CT scans with color-coding.
● Focus on user adoption and workflow integration. Technology should integrate
smoothly into existing clinical processes. Success is evident when clinicians actively
request the technology, as happened when radiologists wanted the AI tool installed on
their personal computers after seeing its value.
2. The Clinician’s Role in a Health Tech Team
● Physicians are essential for meaningful healthcare technology development.
Technology specialists often don’t understand the complexity of medical care, ethical
considerations, and patient relationships. Example: A CEO without healthcare
background assumed a UTI consultation could be reduced to under one minute, not
understanding that even simple cases involve complex patient contexts.
● Clinicians serve multiple critical roles in technology development: domain experts for
clinical functionality, gatekeepers for patient safety, evaluators of technology fit, and
advocates for ethical implementation. Their insight is crucial from concept through
deployment for successful integration.
● Create effective multidisciplinary teams by bringing together medical specialists, data
scientists, engineers, and IT experts. Develop structured evaluation systems to assess
technologies from multiple perspectives (clinical value, integration feasibility, workflow
impact).
● Bridge the gap between startup speed and healthcare institution caution by identifying
and modifying bureaucratic barriers. Example: Reducing contract length from 60+
pages to 3 pages facilitated startup relationships and reduced legal costs. Provide
project managers to help companies navigate hospital systems and regulatory
processes.
● Clinicians must be assertive about their value in technology development. Companies
that fail to incorporate physician perspectives often struggle to create successful
healthcare solutions. Physician involvement is essential for understanding clinical
thinking and workflow integration.
3. Ensuring Safety, Ethics, and Trust
● Patient safety is non-negotiable. Always follow proper regulatory processes (IRB, FDA)
for any technology involving patient care. Run comparative studies to demonstrate
effectiveness and safety. Example: AI tools for detecting lung nodules in ER chest X-rays
reduced missed diagnoses and related lawsuits by automatically flagging concerning
images.
● Maintain the human-in-the-loop approach where AI makes suggestions but clinicians
maintain control and oversight. Physicians must remain responsible for final decisions in
all patient care situations. Key evaluation question: “Would I use this AI tool for a family
member? If not, it’s not ready for implementation.”
● Ensure transparency and explainability in AI systems. Understand what data was used to
train algorithms and be alert to “black box” solutions that can’t be explained.
Technology must be continuously monitored for algorithmic drift (when algorithms
change behavior over time) and bias.
● Address accountability considerations. AI can both reduce liability (by catching issues
humans might miss) and create new concerns. Technology must be continuously
monitored, and institutions should establish clear lines of responsibility for AI-assisted
decisions.
● Design for equity and ethical implementation by vigilantly monitoring for biases in data
and algorithms, ensuring technology works for diverse populations, and implementing
proper data handling and privacy protections.
Future Directions
● Medical education must evolve to include technology evaluation skills for future
healthcare professionals. The goal is not to teach every technology (impossible given
rapid changes) but to develop critical thinking about technology implementation.
● AI can enhance patient-centered care by reducing administrative documentation
burden, transcribing and summarizing information, enabling pre-visit symptom
checking, and supporting personalized medicine approaches through better data
analysis.
Conclusion
AI in medicine is not something being done to clinicians but must be built with their active
involvement. Healthcare professional expertise, caution, and vision are critical for ensuring
these technologies enhance patient care rather than compromise it. The successful integration
of AI into healthcare requires ongoing clinician involvement, careful validation, and a
commitment to maintaining human judgment and oversight throughout the process.