Agile Project Management in Analysis and Reporting of Late Stage Clinical Trials

PharmaSUG 2021 - Paper SI-083

Agile Project Management in Analysis and Reporting of Late Stage Clinical

Trials

Sarad Nepal, Uday Preetham Palukuru, Peikun Wu, Madhusudhan Ginnaram, Ruchitbhai Patel, Abhilash

Vasu Chimbirithy, Changhong Shi, Yilong Zhang, Merck & Co., Inc., Kenilworth, NJ, USA.

ABSTRACT

Analysis and reporting (A&R) in clinical trials requires careful planning especially when multiple projects

are ongoing at the same time in a clinical trial. The current practice typically uses a waterfall project

management structure that assumes one phase to be completed before another can start. When a clinical

trial nears data base lock, the time demands from statisticians and programmers increases due to both

planned and unplanned activities. Clinical study reports (CSR) that are required to be delivered in a short

period of time after database lock also contribute to the increase in challenges and time put towards the

project. Several agile management tools such as Confluence and Jira can be used to streamline the

project plan in an agile way. In this paper, we present details of how agile project management tools

were utilized to enhance team collaboration and ensure timely project delivery. We demonstrate this idea

using a recently completed clinical trial that involved three studies which were ongoing in

parallel and overlapping database lock and CSR delivery dates. A&R team comprising of several

programmers and statisticians collaborated and worked efficiently to deliver five CSRs and three

submission packages in short period of time. These deliverables were successfully submitted to multiple

regulatory agencies without any technical issues reported.

INTRODUCTION

Analysis and Reporting (A&R) is a key step in clinical research process. It requires careful planning to

generate all A&R deliverables and meet required timelines. It becomes challenging when multiple projects

are ongoing with overlapping timelines. All A&R programming done in clinical trials can be divided into four

system development lifecycle phases (SDLC). The Figure 1 provides an overview of these 4 phases.

Figure 1. The 4 phases of system development lifecycle in A&R programming in clinical trials.

There are number of project management techniques that are used routinely in late stage clinical trial

A&R. Of these project management techniques, the two most used techniques are Waterfall Project

Management (WPM) and Agile Project Management (APM). WPM is a more traditional approach that has

been successfully used in the industry for a long time. APM is relatively new development and is being

considered by the industry as a viable replacement to waterfall (Hass, 2007).

WPM assumes that there are predictable events affecting the project and the solutions to counter any

issues arising within the project lifecycle are already present. This project management technique allows

for a project to be split into multiple fixed phases, with each phase requiring the completion of the

previous phase (Bogdan-Alexandru et.al., 2019). In clinical trials, these phases can be broadly classified

as define, develop, validate and operation phase shown in the Figure 1.

The define phase is the most critical phase in the clinical trial development lifecycle as the specifications

for deliverables and resources needed to generate the deliverables are gathered and documented. Lead

programmer and statistician setup meeting with programming team to discuss the programming and

analysis requirements for the planned deliverables. Lead programmer in consultation with lead statistician

creates tasks related to deliverables in available tracking tool and assigns tasks (define, develop,

validate) to individual team members.

After the specifications are developed and individual deliverables task are assigned, the programming

development occurs in the develop phase. Lead programmer in consultation with lead statistician obtains

data as needed for program development as well as sets up the validation plan to ensure quality of

individual deliverables.

Programs developed in the develop phase are validated according to validation plan using requirement

specifications and validation plan in the validate phase. The programs are validated to achieve the

required quality and compliance of available standard procedures. The lead programmer in consultation

with lead statistician reviews completed deliverables and conducts test run of deliverables to ensure

quality. Operations phase is the final phase where the validated programs are executed with production

data to generate final version of required deliverables. Any changes or maintenance is done in this phase

as needed to address new requirements or issues. Lead programmer and statistician review the

deliverables generated by programming team in production area and setup meeting with requestors about

availability of deliverables and completion of programming activities. If changes are required, then

requestors communicate with lead programmer and specifications are updated as required.

The current practice typically uses a WPM structure that assumes one phase to be completed before

another can start. When a clinical trial nears data base lock, the time demands from statisticians and

programmers increases due to both planned and unplanned activities. Clinical study reports (CSR) that

are required to be delivered in a short period of time after database lock also contribute to the increase in

challenges and time put towards the project.

APM could be an alternative and efficient way to manage multiple ongoing clinical trials. APM is not one

specific method but an umbrella term for a collection of methods that share common characteristics. This

project management technique involves the active participation of both developers and project

stakeholders to identify the requirements as well as prioritize functionality of different phases of the

project. This technique is highly effective when the end goal of project is clear, and the customer actively

participate throughout the process (Hass, 2007 and Palmquist et.al., 2013).

Several agile management tools such as Confluence and Jira can be used to streamline the project plan

in an agile way. In this paper, we present details of how APM tools were utilized to enhance team

collaboration and ensure timely project delivery. We demonstrate this idea using a recently completed

clinical trial that involved three studies which were ongoing in parallel and overlapping database lock and

CSR delivery dates. A&R team comprising of several programmers and statisticians collaborated and

worked efficiently to deliver five CSRs and three submission packages in a short period of time. These

deliverables were successfully submitted to multiple regulatory agencies without any technical issues

reported.

MOTIVATION

Our company decided to design and implement a phase 3 clinical research trial. The results of the trial

were planned to be filed with multiple regulatory agencies regarding the outcome of the trials. Among this

large clinical trial, there were three clinical studies, which were ongoing in parallel and had overlapping

database lock and CSR delivery dates. Assimilation of all phase 3 studies data were required for pooled

data analyses in support of Common Technical Document (CTD) authoring for regulatory submissions.

The clinical trial was complex in scope due to the number of studies involved as well as the pooled data

analysis required in support of regulatory submissions. These studies all required individual clinical study

reports (CSR) at the end of each study completion as well as two Integrated Summary of Safety/Efficacy

(ISS/ISE) for pooled data analysis. These CSRs and ISS/ISE timelines overlapped significantly due to the

different start and end time of the individual studies. In total, there were 3 individual studies (P001, P002,

P003) and 2 pooled data analysis. The first pooled data analysis combined all three studies

(P001+P002+P003), the second pooled data analysis combined the last two studies (P002+P003). This

submission package contains 3 CSRs and 2 ISS/ISE. All these reports/summaries also required

associated electronic submission (esub) deliverables in preparation of regulatory submissions in

accordance with CTD guidelines.

The clinical trial followed the Clinical Data Interchange Standards Consortium (CDISC) guidelines for

clinical data including Study Data Tabulation Model (SDTM) v3.1.3 and Analysis Data Model (ADaM)

v1.0. For the A&R team this required generating ADaM datasets, Tables, Listings and Figures (TLFs) for

analysis purposes. The team also needed to generate define.xml, Analysis Data Reviewers Guide

(ADRG), Clinical Study Data Reviewer Guide (CSDRG), Analysis Result Metadata (ARM) and transport

(xpt) files for submission purposes. The total and individual numbers of deliverables of each

report/summary is summarized in Table 1. In order to generate all the deliverables described in Table 1,

the A&R team was assigned 5 statisticians and 10 programmers, including 1 lead statistician and 1 lead

programmer in-charge of the whole clinical trial.

Study/Pooled Data Analysis

ADaM Datasets

TLFs

P001 CSR

197

P002 CSR

200

P003 CSR

199

P002+P003 ISS/ISE

231

P001+P002+P003 ISS/ISE

222

Total

1049

Table 1. Number of deliverables associated with each individual study and pooled data analysis of

clinical trial.

The completion of A&R activities for the clinical program took approximately 18 months. There were

significant overlaps between the individual studies and pooled data analysis especially during the final

phases of report generation. The various phases in development of deliverables were planned in

Confluence and summarized in Figure 2.

Figure 2. Clinical trial A&R activities timeline by study/pooled data analysis.

AGILE MANAGEMENT IN CLINICAL TRIAL

The total number of deliverables required for analysis/submission purposes for the entire clinical trial was

daunting in nature due to the complexity of analysis. This was also true for the individual study or pooled

data analysis. The number of deliverables required was also compounded by the requirement to follow

study data standard guidelines for submissions. The guidelines specify several procedures to follow

ensuring conformance across submission packages. This increase in workload creates a greater

procedural burden than that for a non-submission package. The assignment of work priorities and

resource planning was also complicated due to the large A&R team assigned to the trial. The large team

size also increased the complexity in review of progress on both micro and macro levels.

To efficiently manage the complicated clinical trial, we adopted the agile approach for resource planning

and work assignment. The agile approach consists of iterative planning, development and delivery cycles

which allows a project team to constantly evaluate the deliverables and obtain immediate feedback from

stakeholders. After each successive cycle, the team learns and improves the deliverables as well as their

implementation methodologies. As with WPM, the agile approach requires a requirement definitions and

solutions design phase. Once the requirement at design phase is completed the development and testing

phases are implemented concurrently, and the feedback from the developers or testers are continuously

used to refine the requirements in the design phase. This allows the team to make immediate

modifications to the project requirements or resources allocation based on feedbacks from users as well

as customers. This iterative cycle ends with delivery of projects to customer once all feedback and

improvements are completed. The differences between the WPM and APM development lifecycles are

better illustrated in Figure 3.

Figure 3. The phases within development lifecycle of WPM and APM.

A&R activities in late stage clinical trials are both complex and demanding in nature due to the number of

different phases involved in the trial as well as the number of personnel from different functional areas. A

traditional WPM approach would only work for simpler A&R activities where the number of deliverables or

phases are few and well defined. For trials that encompass multiple deliverable packages or combination

of studies, the use of WPM would be cumbersome as the specification or requirements as well as

timelines would be in flux. The APM with its greater flexibility and collaboration provides a better solution

to the A&R activities. In APM, the project requirements continue to be refined based on the feedback from

stakeholders. It also incorporates flexibility with respect to timeline management. The use of APM also

helps in cost reduction compared to the WPM as making changes to a small part is cost effective.

In our clinical trial, we used Jira and Confluence to implement Agile management approach. Jira is a

project management tool that can be configured to fit any type of project. Team can start with the project

template provided by Jira or create their own custom workflow. Jira issues, also known as tasks, track

each piece of work that needs to pass through the workflow steps in order to be completed.

Administrators with customizable permissions determine who are assigned the issues. With all the

information in centralized location, reports can be generated to track progress and productivity of the

project and ensure timely completion of all tasks. Jira allows user to create a scrum board for APM where

the larger tasks are divided into short and small tasks to achieve productivity. This offer transparency

across project and status of every work item which enables teams to closely monitor the productivity over

time (Atlassian, 2021).

Confluence is a collaboration tool used to map out the project plan, milestones and clarify roles and

responsibilities within a project. In Confluence all the content lives in a dynamic document called a page.

Project plan, milestones, roles and responsibilities and other items of interest can be added in this page.

Teams can start with available project templates provided by Confluence or create their own using

available customization. It can be used with teams of any size and enables communication internally in

open and transparent way. The dynamic ability of Confluence pages gives a team a place to create,

capture and collaborate on project roadmap. Using Confluence, a team can make quick decisions, align

priorities and accomplish project milestones efficiently (Atlassian, 2021).

Jira in concert with Confluence enables teams to apply APM techniques efficiently. This is especially

useful in the context of late stage clinical trials where projects are often complex and have overlapping

timelines. The use of Jira and Confluence in late stage clinical trials helps in efficient allocation of

resources and improved communication about issues and potential solutions across team members in a

transparent manner. In order to efficiently manage all the A&R activities, a Jira project was created by the

lead statistician and lead programmer. This Jira project was setup after the preliminary define

phase described in the Introduction section was completed. The development and validation environment

for creating A&R deliverables was also setup prior to the Jira project creation.

The lead statistician and lead programmer were assigned as the project leads/administrators on the Jira

project. Once the project was setup the lead programmer assigned user access to the other personnel

assigned to the A&R activities. The users in the project could create/browse/search the issues within the

project. Only administrators could access the project administration section and manage user permissions

thereby limiting access to the project. The lead programmer in consultation with lead statistician also sets

up the data classification for the project to curtail any issues with data sensitivity. At the time of the

setup, all team members were asked to only discuss general trial information on Jira and refrain from

using any sensitive or confidential information within the Jira board. The changes made by the

administrator of the Jira project were automatically communicated to the users assigned to the projects

via email notification by Jira. A representative Jira board with weekly sprint and backlog sprint is shown in

Figure 4.

Figure 4. Jira board with weekly sprint and backlog sprint of the assigned tasks in the clinical trial.

Once the Jira project was setup, the lead statistician in consultation with lead programmer sets up epics.

Epics in Jira are a way to classify various sub-projects within a project. The epics corresponded to each

individual study or pooled analysis. The lead programmer in consultation with the programming and

statistician team members defines various tasks within each epic. The tasks within each epic are called

as issues within Jira. Based on feedback obtained during development and validation phases, additional

issues or modification of the existing issues can be done throughout the life of the project. Jira allows user

to import issues in batch. The lead programmer consolidates all the issues within a .csv file that is

then directly imported into Jira for assignment as issues within each epic. The .csv file contains inputs for

Issue Type, Summary, Epic, Priority, Due date and any other relevant fields. The imported issues are

automatically validated by Jira to ensure compatibility with Jira board. The Figure 4 also shows some of

the Epics and tasks that were setup in the clinical trial.

Once the issues are imported or created in Jira, the lead programmer in consultation with the lead

statistician then assigns tasks to individual team member. The lead programmer can also designate one

programmer as lead for each epic in order to efficiently track the progress of issues within each epic. The

issues within each epic can be assigned to a sprint within Jira. The sprint allows for grouping of issues

into a specific timeframe for the resolution of the issues. The concept of sprint can help the whole team

focus on specific tasks within a short period of time and identify gaps earlier. If an issue is not completed

in the current sprint, it can be assigned to a backlog. Backlogs in Jira help the trial leads to identify issues

that are lagging and assign priority to the completion of those issues. The priority of an issue from low to

medium to high can be assigned based on the requirements of the project. The sprints and backlogs in

the project were managed by the leads during a weekly project alignment meeting with the entire A&R

team. During the weekly project alignment meeting, each study member would report on the status of

open issues assigned to them and discuss bottlenecks and challenges. This weekly alignment meeting

ensured the smooth progress of the trial. This also enabled constant communication within the team on

the various lags or leads in the timeline of the trial as it progressed.

During the lifecycle of the clinical trial, the lead programmer and statistician participate in separate

weekly resource planning meeting to discuss the overall status of the trial. Jira reports about each epic,

issues assigned to users and resolution percentage of issues are generated by the lead programmer and

discussed with the lead statistician in these status meeting. This allows the trial leads to identify gaps in

task progression for the overall trial as well as communicate efficiently to the end customers about the

progression of the trial. The use of the Jira report on different sprint and backlogs also helps the trial

leads identify gaps in resource allocation of issues. It also allows trial leads to justify if additional

resources need to be assigned to the trial to ensure timely deliverables of the trial. The various tasks

assigned in sprints and epics of the clinical trial are illustrated in Jira report shown in Figure 5.

Figure 5. Jira report of various sprints and epics in the clinical trial.

VISUALIZATION OF CLINICAL TRIAL PROGRESS

The lead programmer and statistician are responsible for tracking the progress of the clinical trial

throughout the lifecycle of the trial. Using APM and Jira, various reports can be generated that help the

lead programmer and statistician visualize the progress of the clinical trial. These reports are highly

customizable and can be shared with the team in easy to understand formats. One of the reports

constantly used during the lifecycle of the clinical trial previously described was the distribution of total

tasks assigned to individual resources within the trial. Figure 6 shows a pie chart of all the tasks assigned

to the individual resources in Jira. This chart was communicated to internal and external team members

as a representation of the workload of the individual resource. This helped in reallocation of task within

the resources to achieve higher efficiency.

Figure 6. Jira report of total tasks assigned to individual resource assigned to a clinical trial.

Another Jira report that was used extensively in the clinical trial is shown in Figure 7. This created vs

resolved issue report was useful in tracking issue resolution over time. The report helped the lead

programmer and statistician make informed choices about the resources allocated to the trial and

potential gaps that need to be addressed. As can be seen from the Figure 7 there was a significant gap in

created vs. resolved at the time when the timelines for different deliverables were overlapping. But the

allocation of additional resources and changing priority of tasks assigned to individual resource helped

bridge the gap and ensured timely completion of the trial.

Figure 7. Jira report of total tasks created vs resolved in clinical trial over time.

Another commonly used Jira report to track the progress of the clinical trial was the cumulative flow

diagram as shown in Figure 8. This report helps the lead statistician and the lead programmer to visualize

the status of tasks assigned over time. This enables the tracking of trial completion and helps the lead

programmer and statistician in adjusting the priority of tasks in individual epic. As can be seen from the

figure there was a significant gap between tasks that were yet to be started (To Do) vs the tasks that were

completed (Done). This gap narrows and disappears towards the end of the trial when all tasks were

started and completed in the assigned timeline.

Figure 8. Jira report of issue status over time in clinical trial.

CONCLUSION

A&R activities involved in clinical trial development can be complex. This requires close collaboration of

statisticians and statistical programmers with clinical scientist, medical writer and data management, to

generate high quality CSRs and other deliverables. Although each organization might have a

methodology in place to plan and track the deliverables, most of the approaches involved traditional

waterfall management techniques. In this scenario, the lead statistician and lead programming might get

overwhelmed if there are multiple studies within a trial with overlapping timelines and multiple

programmers and statisticians associated with the trial. Additionally, with the advancement of outsourcing

deliverables to Contract Research Organizations (CRO) and location of resources in different

geographical areas, tracking individual progress on deliverables gets really challenging.

We illustrate how to use APM approach and tools in a real clinical trial to help alleviate burden on the lead

programmer and lead statistician and ensure transparency of the trial progress to every study team

member. The use of Jira and Confluence to track the progress of the trial both at a macro and a micro

level helped the lead programmer and lead statistician to efficiently manage the timely delivery in high-

quality. The workflow described in this paper also helped the individual team members increase their

contribution towards the trial completion by enabling transparent communication within the team as well

as tracking overall completion of assigned tasks. The use of epics and sprints within the clinical

trial helped in breaking down a complex project into a series of simple tasks with defined timelines for

each task. This helped the team members to better manage their priorities on the assigned task, as well

as helped the lead statistician and lead programmer by reducing micromanagement of individual team

member regarding a task. The use of epics and sprints also helped in refining of deliverables or

requirements when issues were found without impacting the timeline or deliverables of the entire trial.

The use of Confluence page helped the team members to accurately track the completion of various

milestones in the trial and manage resource allocation to deliverables that were lagging. In conclusion,

the use of APM technique in concert with Jira and Confluence helped in smooth progression of a complex

trial through the various phases of A&R activities. They also helped in reducing communication cost

and time demand on lead statistician and programmer for resource management and task allocation,

thereby increasing efficiency in managing the overall progress of the trial.

Although use of Jira and Confluence to implement APM in this clinical trial helped in the timely completion

of the trial, there are some avenues for improvement. In the future, we plan to implement a workflow to

automate Jira assignment using inhouse tools thereby further reducing time demands on lead statistician

and lead programmer for task assignments. We also plan to integrate cross functional teams including

clinical and medical writing teams into the A&R activities by allowing access to the Jira board and

Confluence pages, thereby allowing seamless communication within the entire trial team. Additionally, we

plan to expand the use of Jira and Confluence in internal standard macro package development.

REFERENCES

Atlassian. (2021, Jan 1). Confluence Basics. Retrieved Feb 18, 2021, from https://www.atlassian.com:

https://www.atlassian.com/software/confluence/guides/get-started/confluence-overview#hosting-options

Atlassian. (2021, Jan 1). What is Jira used for? Retrieved Feb 18, 2021, from https://www.atlassian.com/:

https://www.atlassian.com/software/jira/guides/use-cases/what-is-jira-used-for#Jira-for-requirements-&-

test-case-management

Bogdan-Alexandru, A., Andrei-Cosmin, C.-P., Sorin-Catalin, G., & Costin-Anton, B. (2019, May). A Study

on Using Waterfall and Agile Methods in Software Project Management. Journal of Information Systems

and Operations Management, 13(1), 125-135. Retrieved Feb 18, 2021

Hass, K. B. (2007, May). The Blending of Traditional and Agile Project Management. PM World Today,

IX(V), 1-8. Retrieved Feb 2021

Palmquist, S., Lapham, M. A., Miller, S., Chick, T., & Ozkaya, I. (2013, Oct). Parallel Worlds: Agile and

Waterfall. Technical Report CMUSEI-13-TN-21, 1-101.

ACKNOWLEDGMENTS

The authors would like to thank management teams from Merck & Co., Inc., Kenilworth, NJ, USA, for their

advice on this paper/presentation.

CONTACT INFORMATION

Your comments and questions are valued and encouraged. Contact the authors at:

Sarad Nepal

Merck & Co., Inc., Kenilworth, NJ, USA

E-mail: sarad.nepal@merck.com

Uday Preetham Palukuru, Ph.D.

Merck & Co., Inc., Kenilworth, NJ, USA

E-mail: preetham.paluku[email protected]

Yilong Zhang, Ph.D.

Merck & Co., Inc., Kenilworth, NJ, USA

E-mail: yilong.zhang@merck.com

Any brand and product names are trademarks of their respective companies.