So you are thinking about trying something new, but you aren't sure how people will react? Innovation is risky, but it shouldn't be a dirty word.

Hundreds of questions can emerge when debating a new technology for the office. But the two biggest questions are:

  1. Do we really need this?
  2. Will people use this?

Enter a conference rooms pilots. These exercises are often a great way to get insight on both of these questions. If done right, you'll be equipped with both the quantitative and qualitative information to back up your decision to move forward - or hold back - purchasing a new technology.

What a conference room pilot is (and isn't)

A conference room pilot - or "CRP" - is a helpful practice when it comes to evaluating technology:

"The purpose of the conference room pilot is to validate a software application against the business processes of end-users of the software, by allowing end-users to use the software to carry out typical or key business processes using the new software."

In short, you are giving users an opportunity to see whether the new software will truly help participants with their jobs.

A conference room pilot isn't a demo. While a demo is designed to introduce users to a new technology through an "on rails" presentation, a CRP turns the keys over to users to experience everything for themselves.

A conference room pilot also isn't a trial - an extended test drive without much guidance or engagement. Too often, a trial period yields little helpful information: Was the software used? Did the user understand the goals of the software? What specific features worked or didn't? And so on...

The lessons learned from CRPs tend to be much more immediate. If done right, you will foster an atmosphere of exploration and learning that yields actionable results.

5 keys to a successful conference room pilot

To make sure your next CRP is a success, here's some key things we've learned while doing pilots of ThreadKM:

1. Build the right audience. Selecting the participants is critical. You want to hear from representatives in different parts of the organization, but you also want some attendees who regularly work together. We've hosted CRPs at law firms where key partners were not in the room. While the associates loved our product, they couldn't commit to using it without buy-in from their partners. So the lesson? Make sure all stakeholders are present!

Aim for about 8-15 attendees. Fewer than that means you might be missing out on important feedback. More than 15 people, however, becomes difficult to manage.

2. Start with a Survey Before you can measure the potential impact of the software, you need to establish a baseline first. Find out where the pain points are and how participants are currently trying to solve them.

A survey also puts users the right frame of mind. The questions tee up issues to focus on and give you better feedback during the exercise about whether the technology can really solve the problem.

You can quickly and easily create a survey using online tools like Typeform (my favorite) or Surveymonkey. Make the questions fun - this isn't scientific! Here's a few examples of questions we ask before CRPs:

  • Just guessing, how many emails are in your inbox right now?
  • On average, about how many matters or projects do you work on per week?
  • What is the primary way you keep track of tasks for a project?

Once you've collected the responses, don't leave everyone in the dark! Put the survey results up on the big screen and start reviewing them out loud. It's fun, informative, and a great way to open up the conversation.

3. Let users work in a realistic world. Like having the right audience, a realistic test environment fosters a more productive CRP. Create actual user profiles for each attendee - add profile pics, if possible - to give each a sense of ownership in the pilot.

Then build real-world scenarios for the users to work on. This may take some advance scouting, but the legwork is worth it. You should be populating the software with simulated information that the user would readily understand. In our law firm CRPs, we create workspaces for practice groups, "associate only" and "partner only" threads for private conversations, and project boards for litigation and corporate work. So the content is familiar to the users, even if the software is not.

Next, put together the "script." The script walks through the actual tasks you want users to do inside the software. These should correspond to work they already do - only now we want them to try and use the software. With ThreadKM, we ask people to message each other, share files, create tasks, etc. Invite them to talk out their progress and consult with others on how best to use the software.

4. Watch what users are doing. As an organizer, don't hide behind your laptop. Get up and walk around to see how people are adjusting to the software, as well as if any users are stuck.

For an even deeper view, record the screens of users to playback the results afterward. Or ask your vendor to install software like Inspectlet or Fullstory to review what happened. These record the interactions of users, letting you see how well (or not) they navigate the interface. Like this:

(The red dots above indicate clicks.)

5. Take an exit survey. After everyone has finished the script, it's time to collect feedback. Circulate a post-pilot survey to everyone. This time around, you probably won't be sharing the results. But the information you collect is invaluable.

Qualitative feedback is great: we like to solicit open-ended responses to let the users give input on what struck them the most - good and bad. Quantitative questions, meanwhile, are ideal for summary metrics:

And that's it! Good luck with your own conference room pilot...if you have suggestions for your own pilot, share them in the comments below or email me at!