Evaluating Reporting Solutions Starts with Reporting Needs

How often are you in the market for HR/TA technology without reporting requirements – where you have no need to see or understand or evaluate the data collected and housed in the solution?

Hopefully, your answer is “never”.

Data and reporting are critical for understanding the health and potential of your systems and operations. Without the data, you have no insight into your system. You have no idea if everything is working, or if it’s not, where the problem lurks. You have no sense of opportunities or the ability to validate suspicions. Without data, you can’t answer even the simplest of questions from your boss or executives.

Data and reporting are critical.

And yet, data requirements are so commonly severely lacking.

For example, “the system must have reporting and analytics”, or “the system must have easy reporting”, or “can report on key metrics such as X, Y, and Z”.

Requirements like these are analogous to booking a vacation with the request “I want to go somewhere with water”.

Umm… okay… are you talking an ocean, river, swimming pool, sink?

It’s ridiculous! There’s not nearly enough information to make a meaningful selection. And neither is “the system must have reporting and analytics”.

What about this requirement? “The system must generate the same reports I have now – see attached for the 300 reports we have in our current system.”

So – you want to spend a ton of money, resources and time to do what you already do?

You wouldn’t buy a new computer or car or house that is exactly the same as what you already own, right? Neither should you buy software that replicates what you already have. Such tactics not only waste money and time, but also waste opportunity to update, upgrade, and improve.

To effectively evaluate technology for if it will meet your data and analytics needs, “reporting and analytics” is just the header. The value comes in what’s beneath.

How do build out the full requirement?

Great question!

Here’s the answer:

Step 1: Define it
Have you ever logged into your ATS or HRIS, pulled up the reports or graphs and sat back with a hot cup of coffee to enjoy a good read just for the sake of entertainment? Probably not. Good system reports are not entertainment (though they can be entertaining) or to decorate a slide in a monthly report. They are there for a reason.

To be valuable, Metrics and their corresponding reports and graphs and tables must drive change. This means they do one or more of the following:

• • Drive decision: Giving the organization insight on where to focus, what to fund, and what to set as strategic goals.
• • Drive performance: Providing the organization insight into the health of its operations – what’s going well, what is not going well, and what requires change.
• • Drive behavior: Providing the organization with insight to the performance of its people, who to reward, and who requires corrective action.

To design these valuable reports:

1. Start with the ‘what’ & ‘so what’: What is the question you must answer and why is it important for the organization.
2. Identify the how: how will the question be answered – what data do you need and where does it come from
3. Define the ‘now what’: What is the trigger point for action and what is the action. For example, if a value gets too low or too high, what will the team and/or organization do?

For example:

1. The what: “Are we getting enough qualified candidates applying to our jobs so that we have at least one qualified candidate within two weeks of posting?” The why: we need qualified candidates quickly so we can fill positions efficiently; otherwise the cost of acquisition and lost opportunity gets too high.
2. The how: for each job, track the number of candidates each day that apply and if each is qualified. To calculate how much time has passed between job post and candidate apply, subtract the post date from the date the candidate applied.
3. The ‘now what’: if it takes longer than two weeks to get the first qualified candidate, launch an evaluation of the position to search for potential improvement. For example, evaluate the sourcing and sourcing spend, qualifications, and hiring team availability and responsiveness. If over 10% of jobs take more than two weeks to have a qualified candidate, consider launching an evaluation on the overall hiring process.

Step 2: Mock it
A valuable technique among software and system designers is to build models or mockups of their products and solutions before spending the time and money to build them. These models allow them to test their designs to make sure everything is working the way they imagine, to identify and fix potential problems before they happen, and to verify they are meeting customer needs and expectations.

Designers avoid wasting countless hours, money, and sleep by taking this step. So should you.

Before asking for those important reports you think you and your organization need, build the models. Create mock reports that mimic what you want. Make fake graphics and tables populated with fake data.

To do this:

1. Start with the data: Using the ‘how’ of the previous step, create a list of every data element you will need.
2. Find the source: For every data element, identify where it comes from and/or is stored. This could be a system (HRIS, ATS), it could be from manual steps such as a recruiter filling in a piece of information, it could be from external data sources such as pay scales or population data, or it could be untracked (in which case, part of the requirement will be to begin tracking the information).
3. Draft the report: Create a table with the columns being the data elements outlined in the first step and the rows being the data itself. Fill the table in with fake data. Make sure it is reasonable enough that it will show a realistic example of what the table will look like.
4. Design the graphics: use the table of fake data to make graphs or charts or whatever visualizations you want for your report.

When reviewing your mock reports, do your reports easily answer the question? Can they answer follow-up questions if something doesn’t look right? If not, what additional information should you add to your report to easily answer the follow-up questions?

For example: For the question “Are we getting enough qualified candidates applying to our jobs so that we have at least one qualified candidate within two weeks of posting?”

1) Data: job, post date, candidate, apply date, qualified

2) Source: ATS for all

3) Report: create table

4) Graphics: show a simple graphic to answer if we are meeting the goal or not

‘Mock it’ is particularly valuable when evaluating advanced data and analytics. Rather than trying to explain what you want or getting lost in the marketing claims during a sales process, using a mocked report will help anchor your team to ensure that you are getting what you need.

For example, consider a predictive analytics solution designed to predict the number of hires by role / rank. With a mock of the results, you can explore questions such as what drives the predictions (past performance, sales, market trends, etc.), how granular the information can get (location, department, role, skills), and how to explore the error bars (how accurate the data likely is over time).

Step 3: Test it
No matter how good the specs or description, there is nothing that replaces a test drive. Whether taking a new model car out for a spin, or sitting on a couch in a showroom, or walking through a house, opening closets, turning on facets, and visualizing where all your furniture will go. The same goes for your reports.

After generating a mock report complete with data tables, graphics, and other visualizations and information, take it on a test drive.

Start by defining how will you use this report once it’s generated to inform, enable, and decide. If a report isn’t feeding something that results in action, there is no need for it. Therefore, you don’t need to waste your time or money shopping for a solution that generates it.

For example:

• • Will the reports be disseminated to management to help prioritize efforts?
• • Will the graphs be used in monthly presentations to monitor operational health?
• • Will the data be fed into a dashboard application that calculates team bonuses?
• • Will it be used to feed an investigation when something goes wrong?

For each use of the report and/or graphics and data, set up a test drive. Do this by:

1. Identify the stakeholders of the report (the recipients of the information)
2. Ask the stakeholders to participate in the test drive for the purpose of providing feedback and additional requests on the mock report/graphics/data.
3. Present the fake data as you would the real information (report, slides, presentation, etc.)
4. Gather feedback from stakeholders. For example, what they like and not like about the report, what follow-up questions would they have, and what other information do they require to facilitate a decision.

Evaluate the feedback and take action as appropriate to adjust the mock reports. If there are substantial changes, take it for another test drive to verify that it now meets the needs of the stakeholders.

Note: you do not need to implement every feedback item. Concentrate on the feedback that drives understanding and enables action and decisions. Save the rest of the reasonable feedback (i.e. requests that enhance the experience but are not required for success) as ‘nice to have’ requirements for the solution vendors.

Step 4: Use it
Now it’s time to get the huge return on your efforts. Use your mocked and tested reports as part of your requirements when evaluating vendors. Using the mock report as the requirement makes it abundantly clear what you want and how you want it.

For anything that is less obvious or not ‘functional’ in your report, add notes. For example, if you want to click on a report to drill down into more detail or use filters to specify time or department, but those functions aren’t in the mock report, put them in a note.

Now here’s the best part – let the vendors do the heavy lifting from here.

Have the tech vendors tell you what they can and can’t do to generate the report, how it can be done, and how much it will cost. Or, even better, if possible, have them generate a mock report using their system to show you what they can do for you.

Then compare the results between vendors and select the best fit.

Watch for vendors that don’t specify the how or the costs. Many vendors will say yes to just about anything but when it comes time to implement, you discover the detail that it takes extreme customization at a heavy price tag.

If the vendor is vague with the details (i.e. they say “yes”), press for more information on how (out of the box, configuration, customization, with an add-on, etc.) and at what cost (technology, implementation, support).

Bringing it all together
Reports are an extremely valuable tool for any technology system. They provide the ability to see what’s happening behind the pretty screens and background processes. Great reports allow you to drive strategy, performance, and behaviors of your solutions, teams, and programs.

To get truly fantastic reports, start by identifying what you need, mock it up, test it, then use it with vendors so you are getting exactly what you need to drive action and not just a mountain of pretty pictures that do little more than take up space and decorate slides.