The best way to validate the functionality of a technology system is to test it. Most often, testing includes running a pilot or a proof of concept (POC). But what's the difference between a pilot and a POC? How do you decide which is relevant for what project?
The terms "POC" and “pilots” are often used interchangeably by teams deploying technology within legal organizations, but they have different meanings. The opacity generated by indiscriminate use of these terms is not helpful, not least because vendors often have specific license terms that govern validation tests. To understand if the terms are reasonable, the team needs to understand the offerings and whether it makes sense for the type of validation you are seeking to do.
For this piece, LTH interviewed three KM experts to help further explain the differences between a POC and a pilot, and provide guidance on when you might use one over the other.
Defining a POC
A POC is generally considered a less significant test of a technology's functionality than a pilot.
Elisabeth Cappuyns, Director of Knowledge Management at DLA Piper defines a POC as an internal KM or IT process where the team considers how a tool might integrate into the firm. For example, a POC could test just one aspect of the technology. In legal, a POC is often used as a way to prove the use case or the business value of the product – to make sure that the technology being considered is indeed fit for the purpose for which it is intended to be licensed. In other words, does this technology address user needs the way we hope it will?
Patrick Dundas, Knowledge Management Special Counsel at Schulte Roth & Zabel LLP, defines a POC as a formal test with an articulated testing rubric where each participant in the POC has a specific quantitative task to complete and analyze.
During a POC, it’s not unusual for a decision to be made to switch off some of the technology's capabilities. For example, you might not allow users to input data into a system, even though that is how you ultimately want the technology to be used. Instead, you will set up the technology to create the relevant workflows for the use case and ensure that these workflows work as you expect. The workflows in this instance are unlikely to be "live" – they will be set up to mimic how the end-users will work with the technology when it has been implemented. In this sense, a POC is effectively a prototype or a pre-prototype of the final solution.
Patrick further explained that a POC does not always provide a definitive answer as to whether the firm or organization should deploy a technology system - unlike a pilot. For example, a firm could run a POC on an AI due diligence engine to prove that they can get value out of the product. A firm won’t run a pilot, however, until it has established a business case for the technology and a determination that the project will go ahead.
A POC may not even involve end-users, although LTH recommends that end users be involved in all project stages. Instead, a POC might leverage knowledge management, legal operations, IT, or innovation team professionals who test the workflows and record results that indicate whether or not the technology meets requirements. A POC takes as long as is required for the testers to validate the solution. Depending on the technology's complexity, a POC's duration could range from one week to a month.
A POC will usually occur before signing a license agreement with a vendor or under a separate agreement pertaining only to the POC. After the POC is successful, negotiation will emerge as the parties progress towards a license agreement. Patrick noted that he has seen many firms invest substantial resources into the POC to get effective results.
Defining a Pilot
In contrast to a POC, a pilot is a complete test of the technology and can often serve as the early stage of a roll-out or deployment. A pilot is a more sophisticated and controlled form of a POC, and might be set up within a license agreement as an initial "test run" of the technology. In these cases, the arrangement generally provides the organization with a "get out free" option if the pilot doesn't go according to plan.
The pilot is intended to test all aspects of the technology within the relevant live workflows. Simon Wormwell, Chief Strategic Enterprise Initiative Officer at Osler, explained that in a pilot, the internal value of the use case has already been established. Demos have been conducted, a preferred vendor identified, the internal process has been documented, the requirements are precise, and now the next step is to effectively test implementation and evaluate what technology might look like in production. In addition to testing functionality, a pilot is often intended to also test business assumptions about the solution, for example, whether end-users will use the pilot as part of their regular practice.
As a result of the broader requirements of a pilot (compared to a POC), they can be much longer. A pilot will typically last anywhere from a month to six months, though some may run for a year.
A properly conducted pilot will leverage a subset of end-users. The goal is often to ensure that bugs and kinks in the solution are ironed out before a full roll-out to relevant end-users. If the solution is intended to be deployed to all users across an organization, the pilot might be run with just one department. If the solution has a narrower focus, for example, it will serve only one practice group, and the pilot group might be a set of "super users" of technology within that group who can be counted on to provide feedback.
When to Use Which Model of Validation
● You have concluded the demo phase of an AI due diligence software and have interviewed various vendor references. After an informal one-week trial, the team conducts a formal test of the software (or a POC), where each team member is assigned a specific task that is measured and scored in a rubric. An example of a specific task includes how key data points are extracted from structured files.
● You are rolling out a directory product, and you need to ensure the right fields exist to capture the information you need, that inputting information is intuitive, and that the interface is clean. A POC will allow you to vet the details to ensure it is the right platform.
● You are testing software in relation to a process that will undergo a sufficient amount of workflow change if the tool is rolled out. The goal is to test it with 30 to 40 legal professionals eventually. Before using such a large sample group to understand if the system truly performs how it is supposed to, it is tested as a pre-pilot or POC with five legal professionals to validate the core functionality. This test helps the team conclude if they should go further into a pilot to evaluate the tool in a commercial production environment.
● You have decided on the solution that is the best fit for your internal use case, the security team has vetted it, and you have signed a contract with the vendor. As the solution is expensive and the practice group end-users for whom this solution has been identified are particularly change averse, you have negotiated a six month term in the contract that allows for an initial pilot during which adoption can be measured among participants. You intend to use this time to validate the usefulness of the solution to the lawyers it is meant to support. If adoption during the six month pilot is low, you can re-visit the investment with the practice group chair and the vendor to determine whether it’s still worth proceeding.
● A significant investment in a technology platform has been made by the firm, and the technology has been implemented on prem. In order to ensure that the design and configuration elements of the system meet end-user needs, and to identify any bugs before a broader roll-out, you run a pilot with a smaller practice group. During this pilot, you track metrics that enable you to tweak and configure aspects of the technology that will ensure better success when you roll out to the whole firm.
● As part of an adoption strategy, you implement a pilot stage prior to rolling out a new technology to all parts of a practice group. You engage super-users and influential members of the practice in the pilot and leverage their positive feedback to create messaging and communications that can then be used to generate interest and momentum when the technology is rolled out more broadly.
Both pilots and POCs are useful testing mechanisms to validate the functionality of a technology. There are certain circumstances where one might be preferred over another, depending on the complexity of the tool, the number of vendors you are considering, or the available resources. Carefully considering which option is the best fit given your circumstances is one of the first critical steps in successfully adopting a new technology solution, and will ensure you can negotiate the right terms with the vendor.