Healthcare is facing increasing quality and resource challenges and telemedicine is often regarded as part of the solution. Numerous pilots are underway, but will these telemedicine services really make healthcare more efficient? Will they increase access, quality, safety of care? And what added value can satellite communication bring to such services? These are questions the ASSIST SATCOM-APPS study aims to address by developing comprehensive methods and easy to use toolkits for telemedicine assessment, which help pilot projects to objectively and rigorously evaluate their solution from an evidence-based multi-stakeholder perspective.
ASSIST is funded through the ESA’s General Studies Programme (GSP).
The ASSIST documentation and the associated toolkits can be downloaded form the ASSIST web site.
The holistic goal of this SATCOM-APPS study is the development and validation of a comprehensive methodology to evaluate outcomes and impacts generated by ESA-sponsored projects related to the provision of sustainable telemedicine services. The ultimate goal is to provide clear, relevant and impartial information to be used for evidence-based decision making concerning the migration of pilot services into real, operational undertakings.
The evaluation of health information services has to deal with a multitude of heterogeneous variables, such as actors (the people), artefacts (the technology), and the environment in which they are implemented as well as with their interactions and most importantly, their outputs and outcomes. Telemedicine has medical, technical, psycho-social, organisational, business and societal aspects. Assessing telemedicine means to incorporate all these aspects. Research shows that there is a huge bandwidth of evaluation methods, but many - if not most - of them are not very suitable to deal with a comprehensive, holistic evaluation of a telemedicine service. It should be emphasised that a telemedicine system is only part of this solution, whereas the telemedicine service also contains the whole delivery chain and all stakeholders. In addition, it is different from project evaluation which is measuresing the achievement of project objectives, whereas ASSIST is focusing on the impact achieved or achievable.
Selection criteria and critical factors to be considered in this context are the extent to which methods allow for:
- Benefits and costs estimation
- Provisions for testing rigour
- Dealing with objectivity issues and bias in measurements and estimations
- Sustainability analysis: ex-ante and formative assessment, business modelling, affordability considerations, risk assessment
- Multiple stakeholders to be considered
- Health system and operational environment variables to be taken into account
- Time dimensions (from planning to routine services)
- Validity, reliability and comparability of measures and options
- Conciseness of model
- Applicability to telemedicine / space technology
None of the methods available as of today appears to be readily suitable for telemedicine assessment. All lack some elements, sub-methods or techniques required for the purpose at hand. The starting point for the tools developed in the ASSIST project is the EHR Impact cost-benefit analysis approach and method, which provides the most advanced, and differentiated tools and techniques for data gathering and analysis. ASSIST will further develop and extend it with additional techniques to fit the project goals. ASSIST tools will allow a telemedicine project to self-evaluate their project impacts in a multi-dimensional way. It will support the evaluator in finding the data needed and analysing them. A small number of outcome measures allow to easily understanding the relation of benefits and cost. Pie charts allow assessing the distribution of cost and benefits over different stakeholder groups, identifying benefits shifts and the distribution between financial and non-financial benefits. Project leads will receive projections of results which allow them to sketch out scenarios of future use.
The project team will always take the perspective of stakeholders and decision makers, and look at the added value they may receive.
The study plan includes the following parts:
- A critical review of the existing, consolidated and comprehensive methodologies established so far for evaluating telemedicine projects.
- The elaboration of a set of adequate methods that can be adopted to carry out a systematic evaluation of the results of such projects within a multi-stakeholder perspective.
- The development of an evaluation model for the results of such projects, based on the selected assessment framework and its methods, and composed of a set of guidelines, procedures and an associated supporting toolset.
- The application of the evaluation model to a number of space-based telemedicine projects for validating the model, with a subsequent critical revision of the model.
- The delivery, presentation and promotion of the final evaluation model, and dissemination of the associated results.
ASSIST has been closed after successfully achieving its goals. To support project leaders and entrepreneurs practically, the project has created a self-assessment tool that reflects the methodological requirements of the assessment framework. The ASSIST assessment framework utilised the existing experience in the socio-economic evaluation of telemedicine and eHealth by an in-depth analysis of 16 assessment frameworks and study team’s own experience in evaluating more than 50 projects.
The ASSIST assessment framework builds on Cost-Benefit Analysis (CBA). ASSIST has defined 30 stereotypes of stakeholders that are meaningful to telemedicine services. Each stakeholder has its own set of pre-defined indicators covering the most important impacts occurring in telemedicine.
In total ASSIST has defined 339 indicators for its 30 stakeholders. ASSIST calculates several performance measures. They help users identify affordability aspects, financial risks, and points of highest impact. A validation with four telemedicine projects showed that that it can be very beneficial to projects but still the first effort in learning to handle the tool could be reduced.
ASSIST is available for free und a GPL licence from its project website http://www.assist-telemedicine.net/. Future telemedicine projects sponsored by ESA will be encouraged to use the ASSIST tool.