Back to the future… or the amazing lack of progress in effective document review

In 1999, as a still relatively inexperienced medical writer, I was introduced to some software that was going to revolutionise document preparation: Documentum, a collaborative reviewing tool for regulatory documents. These documents would be drafted in a secure environment. Version control would be guaranteed. Workflows would ensure that documents were reviewed in a neatly structured process within defined timeframes. Reviewers could see each other’s comments and respond to them during the review. Collation of comments would be automated. The software would also remind reviewers of their obligations and provide an audit trail of their reviewing activities. This was the way to go. And remember, this was the year 1999.

This brave new world was going to solve the technical inefficiencies involved in document preparation at the time. No more distribution for review by email. No more comments received in multiple documents. Or worse, multiple permutations of comments on top of other people’s comments, needing to be pored over by medical writers to ensure nothing was overlooked. No one would review the wrong version. Medical writers would no longer have to consolidate comments into a single file, this would all happen automatically.

In fact, add a few other ingredients to the process, like training the team in reviewing expectations, and some coordinated planning, and you have a genuine increase in the efficiency of document review. It’s a win-win situation. Documents are reviewed more effectively, quality goes up, and time and cost go down. Surely there isn’t a manager in the industry who can resist that?

Is this how your review process happens? If yes, then you’re probably one of the lucky few. Already back in 1999, with Documentum in place, my first sobering experience was discovering that the company in question had decided to disable the workflow function. So the plethora of separate files with review comments, and the painful and costly process of collating and consolidating comments by hand, continued unabated. But at least there was a functional document repository in place.

Since then, working as a contractor, experiencing the inherent diversity of different companies has been quite an eye-opener. Our clients encompass the entire bandwidth of the industry, from global multinational to biotech start-up. Their document production processes range from the well-structured and technologically advanced to the poorly structured (to put it mildly) and technologically challenged.

Unfortunately, ‘well-planned’ and ‘technologically advanced’ is the exception. ‘Poorly planned’ and ‘technologically challenged’ tends to be the rule. The use of Documentum or any of the other collaborative reviewing tools remains an exception. Note, though, I’m now talking about the year 2015. So what has happened since first encountering the brave new world of document review back in 1999? Apparently, very little in the case of many companies, if not in the majority of them. Since 1999, my company has worked with around 100 different clients of all sizes. Only five of these clients (not all of them ‘big pharma’) actually use the collaborative reviewing tools as intended. Many clients possess the software but don’t use its full capability. This is in effect the same situation as in 1999. Only three of these five clients have a well-structured process integrated with the effective use of a reviewing tool. These clients are truly leading lights in the field of document production. For reasons of confidentiality, they unfortunately have to remain anonymous.

Of course there’s more to document review than a reviewing tool. There is also a need for effective management and planning of resources, which are also frequently lacking. As medical writers, we’re often confronted with the absurd situation of having to prepare documents that are reviewed using less than optimal procedures, even though software solutions are either already available or could be available for little cost.

So given this current situation, it’s worth discussing what it takes for effective document review. I believe there are six vital ingredients to the process, all of which are used effectively by the leading lights mentioned above.

DEFINE A STRUCTURED REVIEW PROCESS

This may sound obvious, but surprisingly often it’s either partially or completely absent. Especially because we operate within a highly regulated world, the obvious approach is to have a standard operating procedure (SOP) or some other written guidance detailing the process step by step. Procedures should be defined for what is to be reviewed, how the review is to be conducted, when, and by whom. These procedures should be pragmatic and non-negotiable. There’s nothing new or even mysterious about the management and planning skills needed for effective document review.

All too often, companies have some form of guidance in place, but it isn’t rigorously applied, some- times even if an SOP is involved (see Enforce the review process, below). When a structured review process is lacking, confusion within the reviewing team is almost guaranteed. Diverse approaches to reviewing can result, and it can be challenging for medical writers to obtain the consensus needed to move documents forward.

As part of a structured review process, a kick-off meeting for any given project should also be used for clarifying timelines in terms of what is realistic, where inter-dependencies and potential threats lie, and mitigation strategies for any potential threats. Agreement and commitment to the timelines is then needed at this stage from all stakeholders, including contractors. Ideally, the timelines should then be actively managed during the course of the project, and agreement and commitment obtained from all stakeholders when changes to the original plan are needed.

USE A COLLABORATIVE REVIEWING

When used correctly, a reviewing tool increases the effectiveness of document review dramatically. The benefits have been listed above, and they translate into true reviewing efficiency. There are a number of such tools on the market, either within the environment of regulatory software such as Documentum or SCORE, or as an online subscription service such as Please Review or, specifically for manuscripts, Datavision or Pubshub.

Of course, there are challenges to implementing a reviewing tool. Often the greatest of these is resistance to change. While some clients have insisted that reviewing tools are impossible to use, others have used them successfully for years. It’s a question of enforcement (see below).

Another perceived challenge is the cost of implementation. This is not because the company cannot afford the software (any company that can afford to develop drugs can also afford the resources for effective planning and a collaborative reviewing tool), but because there’s been a failure to justify the cost in the right places, perhaps exacerbated by the resistance to change. However, no one can seriously claim that it’s more cost-effective to have medical writers manually collate comments from multiple files rather than having the software do it automatically with a medical writing cost of almost zero.

Another challenge is the training required for using a reviewing tool, especially when external collaborators are involved who may also face technical challenges when accessing the software. But such challenges are certainly anything but unsurmountable.

CLARIFYING REVIEWING EXPECTATIONS

It never ceases to amaze me how little attention is given to providing training in reviewing expectations. Everybody involved in writing and reviewing documents should receive mandatory training in how to review documents. This should include training on how to prioritise comments (e.g., major, critical–must be addressed; minor, not critical – can be addressed at the team’s discretion; and cosmetic), how to respond to other reviewers’ comments (assuming a collaborative tool is used), and how reviewers should focus primarily on their own area of expertise. A clinician should not check whether abbreviations have been defined at first use, or invest time in imposing personal linguistic preferences on text that is already linguistically correct.

Another major role of training is to impress upon reviewers the need for constructive, specific, and unambiguous comments. Any changes requested must be specific enough for the medical writer to understand the issue and obtain consensus on whether the change should be implemented. Reviewers must understand that open comments with no specific direction are unhelpful and usually counterproductive. A couple of real-life examples: ‘Can this be phrased more clearly’ (what exactly is meant by ‘more clearly’?); in a study report: ‘Who decided on this study design?’ (the medical writer may or may not know, but this is irrelevant when writing up the results). Or an all-time favourite of mine from a Global Head of Regulatory Affairs, placed nowhere in particular within the final draft of a study report written under extreme time pressure: ‘This is bad’ (did he mean the results? The conclusion? Was the study poorly executed, or was he complaining about the writing?). The man was senior, so this type of comment (he had more of them) couldn’t simply be ignored, but he certainly didn’t understand how to review a document.

A good understanding of reviewing expectations is also needed for resolution meetings. Reviewers should attend such meetings (see Reviewing as a defined activity, below), understanding the need for pragmatic resolution of outstanding issues. It’s important that everyone understands the need to avoid hijacking resolution meetings with specific issues, especially those that are less relevant for the overall document goals. All reviewers must aim to resolve all comments within the framework of the meeting.

IMPLEMENT STAGED REVIEWS

To reduce the writing and reviewing burden during later stages of document preparation, when time will probably be tight, it can be helpful to plan a staged writing and reviewing process. Taking the example of a study report, usually data-independent sections of a document can be written and reviewed ahead of the data becoming available. The aim should be to obtain consensus on these sections without the com- plication of having to think through the results at the same time, and this process can also help with structuring and reporting the results.

Teams usually find staged reviews attractive as they are a promising means of ensuring that timelines are met. While they will readily agree to implement the concept, they often don’t fully comprehend the consequences. These are that the review of these initial sections then really does have to happen in advance in the earlier time slot allotted, and that true commitments in terms of structure and messaging, etc. have to be made at this time.

Assuming that this takes place as planned, the reviewed sections should be ‘locked down’ with the clear understanding that they will not be revisited and reviewed again at a later stage of document preparation. Locking down can be achieved by changing the colour of the text and instructing the team appropriately. This should come as no surprise to the team if a structured review process is in place and everyone understands the processes to be followed (see above).

PLAN REVIEWING AS A DEFINED ACTIVITY

Although managers are loathe to understand this, reviewing documents really does take time. So it is essential that reviewing is planned as a defined activity with realistic and negotiated timelines, including resolution meetings. This reduces the risk of an excess of competing priorities, with reviewers finding themselves having to review multiple complex documents simultaneously.

Practiced unfortunately only rarely, an effective approach is to have reviews scheduled in electronic calendars as all-day appointments for the intended review period. These appointments don’t block reviewers’ calendars, or prevent multiple reviewing requests or workflows being issued, but they do help to visualise the reviewing burden at any given time. Electronic calendars should also be used for generating automated advance notices for impending reviews and appropriate reminders during reviews. If timelines are extremely tight, then blocked reviews on a particular day, as specific activities that exclude competing activities, can be effective.

Another essential component of having reviewing as a defined activity is a defined follow-up procedure for stakeholders who don’t complete their review as planned. Ideally, the overall process should be conducive to reviewers being able to review in the allotted time period, i.e., through the minimisation of competing priorities. However, a culture should prevail from the outset that failing to respect reviewing timelines is inconsiderate to other team members and unacceptable. The extent to which this is enforced is highly variable across companies, but the leading lights enforce this vigorously.

Even under the best of conditions, circumstances will always arise when a reviewer cannot deliver on time. Ultimately, the response to this situation depends on how essential a given reviewer’s input is. Some companies have a ‘no response is agreement’ policy and move on, but this can be problematic when, say, senior management input is needed. But even with such influential reviewers, an agreed-upon strategy for non-responders is needed to enable the team to advance the document to the next stage.

ENFORCE THE REVIEWING PROCESS

Even when a defined review process is available in writing, often the company fails to enforce it, in part or even completely. This can be due to inadequate project leadership or an aspect of corporate culture at the company involved. In the few cases I have witnessed an effective review process, this was because the process had, and was seen to have, management authority at the highest level.

Initial and refresher training events are needed to reinforce reviewing principles and processes, together with management’s expectations that defined procedures are mandatory. In this way, a culture can develop where it is absolutely clear that the reviewing process is not negotiable.

The training must also reinforce the idea that any delay in providing review comments complicates the writing of subsequent drafts, can delay timelines, and disadvantages other stakeholders in the project. Almost every medical writer can recount any number of projects where the lack of reviewing discipline, especially in terms of comments being provided late, has been a substantial challenge. The diversity among companies is quite amazing, but in companies where good reviewing practices and discipline are lacking, there’s often resignation and indifference to the situation, and ‘muddling along’ seems to pervade as an acceptable corporate culture in the absence of empowerment to change the situation.

So, having briefly summarised some of the cornerstones of effective document review, I think it’s clear that many medical writers are, for any number of reasons, still having to write documents within a framework of ineffective document review processes. While medical writers are in many respects best positioned to advise on enhancing the efficiency of document reviews, in some ways we also partly contribute to the problem. This is because we often act as a buffer, enabling teams to meet their timelines despite all the inefficiencies along the way.

The classic situation to illustrate this is how we react to delayed comments. Recently, I was working on a complex document with a team of writers. The client, with no negotiation, suddenly reduced the timelines for the project by 4 weeks, so review cycles had to be compressed. Despite the new situation (or because of it), a key reviewer failed to deliver any comments by the planned deadline. The reviewer then delivered comments in five different Word files staggered over the fol- lowing 2 weeks, and even then didn’t review the entire document (the missing parts were saved for the next draft, making this later review more complex than planned for). The last of these files arrived just a couple of days before the next draft was due to be issued. Besides the time pressure to work through the comments for the next draft, there was no opportunity for team resolution of issues that the writing team couldn’t resolve, so these also had to be saved for the next draft.

All this happened in 2015, and it’s already clear from the brief description of this example that this client is no leading light in terms of its document review process. Most of the vital ingredients described above were lacking. In this type of situation, which is certainly not unique, I’ve often dis- cussed with the client why things are the way they are and what we, as contractors, could do for that specific client. The responses are fascinating. Some clients tell me that certain practices (included in the vital ingredients above) are unenforceable at their company, in which case I reflect on the fact that I’ve experienced exactly these practices enforced and working well at other companies….

Clearly, for quite a number of companies, some aspects of document review really haven’t changed much since 1999. Good organisation and management (the majority of the vital ingredients above) will probably always remain a challenge, implemented with varying degrees of success. But while we otherwise embrace technological advances that can enhance efficiency (there were no smartphones back in 1999, but who doesn’t have one now?), the brave new world of collaborative reviewing tools available even back then has certainly not been fully embraced across the pharmaceutical industry. In this respect, it’s clearly time to go back to the future….

AuthorDouglas Fiebig
JournalMedical Writing, the journal of EMWA
Volume/YearVol. 24, No. 3, September 2015
Download as PDF  (77 KB) Return to Publications