When is a (new) summative evaluation needed?

How design changes impact usability

A lot of medical device manufacturers often wish to update their devices through the product life cycle. This can be software updates or minor aesthetical or functional modifications to the design. However, regardless of the update to be implemented, all changes must undergo an impact assessment in terms of their impact on the safety and effectiveness of the device. This article will introduce you to assessing the usability impact of your design change and thereby plan for any needed summative evaluation. Let’s look at a few examples:

  • An insulin pen is being relaunched with a new drug. The users and the use conditions are unchanged.
  • The manufacturer updates the Intended Use of a medical device to market the device for home use. The device has previously been designed and validated to be used in a hospital environment only by trained healthcare professionals.
  • The screen of an already marketed medical device is changed but no changes are made to the Graphical User Interface, the Use Specification or the Intended Use.

How should these design changes be handled from a Usability Engineering perspective and which activities should be planned when the device is already on the market?

The focus of Usability Engineering for medical devices is safety and effectiveness. Therefore, the evaluation must assess the impact on these parameters. This focus on safety and effectiveness is not unique for changes to the device user interface but are in focus of all design changes. Many manufacturers choose to have a checklist for assessing impact of all design changes. It is also important to document why something does – or does not – impact e.g. the usability engineering file.

Impact assesment Yes No Comment / Rationale
Does the design change impact the Intended Use?
X
No changes to intended use.
Does the design change impact the Use Specification?
X
No changes to user groups, patient groups, use environment, operating principle or indications for use.
Does the design change impact the Labeling changed?
X
No changes to labeling.
Does the design change impact the Design Verification?
X
Design verification must demonstrate that the design input requirements are still met when implementing the new screen.
Does the design change impact the Design Validation?
X
If the technical performance of the screen is verified according to the design input requirements, not further design validation is necessary as the conclusion of the existing design validation is still valid.
Does the design change impact the User Interface (UI)?
X
No changes to the user interface.
Does the design change impact how the device is used?
X
No changes to how the device is used.
Does the design change impact the Requirement Specification?
X
No changes to the requirements specification.
Does the design change impact the Risk Analysis?
X
No changes to the risk management file.

In some cases, a more thorough assessment is needed. Thorough analysis and the involvement of a Usability Engineer is often necessary to assess whether the design changes impact how representative users use the device safely and effectively. Technolution recommends assessing design changes from a Usability Engineering perspective in the following way:

Does the UI change involve a safe-related task?

The first question to answer is whether the design change is related to a critical task, i.e. a task if performed incorrectly – or not performed at all – could cause serious harm to the patient or user. If not, it should also be assessed whether this design change introduces a new risk.

Has the existing UI been evaluated?

If the device has never been evaluated in a summative evaluation and it is related to a critical task, it is recommended to perform a summative evaluation. It is assumed that critical tasks have been evaluated for already marketed devices. If the device has already been evaluated, the conclusion of this evaluation may still be valid.

Does the design change impact safety-related use?

To answer this question, the involvement of an experienced usability professional may be needed. If the design change affects the safe and effective use of the device, a summative evaluation shall be performed as described in both IEC 62366-1:2015 and FDA’s guidance on Human Factors Engineering from 2016. If the design change does not influence how intended users perform critical tasks using the device, a thorough assessment must be documented. Such an assessment may use existing usability data from the previously performed summative evaluation.

Evaluating the impact of design changes on how a device is used safely and effectively

Let us look at a couple of possible outcomes of the impact assessment:

  • If the design change impacts – or there is uncertainty whether it impacts – how a device is used safely and effectively, a summative evaluation must be planned.
  • If the design change is identical to a previously implemented protective measure that has been successfully evaluated, it can be argued that the updated device can still be used safely and effectively by the users. It is recommended to perform an assessment of how equivalent the design change is with the previously implemented protective measure.
  • If a medical device is already successfully evaluated by representative physicians and it is now to be used by nurses as well, it may be possible to argue that these two user profiles have similar capabilities and limitations when using the device. Such argument can be emphasized by performing a formative evaluation that supports this conclusion and did not identify new risks.
  • If the implemented design change ensures that the device can no longer be used incorrectly (i.e. eliminated the risk), a design verification is sufficient. This is based on the principle of inherent safety by design which is known from poka-yoke solutions. For example, the risk of installing an expired consumable in a device can be eliminated by adding a chip containing the expiration date to each consumable. This information is then read by the device to ensure that the consumable is not expired.
  • A change in the labelling or instructions that elaborates on the consequence of storing an ampoule in direct sunlight may not require further summative evaluation. Especially not if previous evaluations have found that the users understand how to correctly store the ampoule and avoiding exposure to direct sunlight. Such argument can be emphasized by performing a formative evaluation that supports this conclusion and did not identify new risks.

We wish you good luck and success in assessing your design changes. Remember to include all aspects in your impact assessment – also Usability Engineering. Please don’t hesitate to contact Technolution for any questions or inquiries.

feel free to contact

Morten Purup Andersen

Senior Development Engineer / HFE Specialist
mpa@techolution.dk
+45 4526 1000
Want to know more?

Insights you might also be interested in:

Tagged