Rating online degree program

Before choosing a product, first ask many questions about how your organization will use multisource feedback.

Today's developers of 360 [degrees] feedback software offer an overwhelming assortment of excellent, highly sophisticated programs for gathering, processing and reporting employee feedback. These full-featured products offer optimal flexibility, resources and power. The programs selected for review include features such as customizable surveys, a library of questionnaire items, multiple reporting options with graphic displays, action planners and implementation procedures. Some also have functions other than performance feedback, such as employee or customer satisfaction surveys.

The programs display inviting, user-friendly screens that dazzle with creative graphics, tool bars, drop-down menus and online help. Prices are lower than they have ever been, and the systems are so affordable that many employers extend their use to all employees. Even with all these positives, some negative traits are bound to show up when you evaluate the programs for your most-needed functions.

Evaluating software capabilities

Three primary, criteria were used to select programs for review. First, the program can be purchased independent of consulting services; second, it can run on a stand-alone PC; and third, it has a base of users for the product reviewed or an earlier one. Four programs meeting those criteria were selected.

CompStar Appraiser Plus 360 from Questar InfoComm of Salt Lake City and 20/20 Insight GOLD from Performance Support Systems Inc. of Newport News, Va., were near completion of beta testing when reviewed. The two systems were scheduled for release early in the fourth quarter 1998 and may be further enhanced in the future. PulseTools, introduced earlier this year by Vitality Alliance of Provo, Utah, is an upgrade and companion module to CorporatePulse. Intelligent Consensus 3.1 from Teams International of Tempe, Ariz., has been on the market since 1994. The company planned to introduce Version 3.2 during fourth quarter 1998, but the beta version was unavailable for review. Teams International is also introducing a service option, TEAMS360.com, which allows companies access to the software on a fee basis through a custom web site.

To help you assess the programs, I compared them with various criteria identified in the following three scenarios for level of need.

A basic, low-cost system. Suppose your company has 300 employees and a budget of $10,000 for this project. You recently set up a workable 360 [degrees] process using paper forms but now need to automate the process and track respondents to administer the system more efficiently. You want to keep things simple and avoid overwhelming employees with extensive reports.

CompStar Appraiser Plus 360 (CompStar) can handle those basic needs within your budget. This solid tool packs in some powerful features for a modest price. It's primarily designed for network use and has the option of exporting surveys for off-site evaluations. A modest library, of core job-behavior questionnaire items is included, along with goal-planning and development modules. Questionnaire items can be fully customized, and you can choose from a variety of rating scales and customize the weighting descriptors. Respondents can insert comments to support their ratings of job behavior, as well as summary comments. The program permits weighting of individual job behaviors, goals and development plans; the weighting can be applied within a respondent category and by type of respondent.

The software currently generates only individual reports containing each respondent's ratings and comments but does not provide a consolidated report. Also lacking are options for graphic display of reports, such as shaded or color-coded horizontal bars, to compare respondent ratings. The individual reports do include a final summary page by respondent category-such as self, peer or supervisor-that lists a consolidation of average feedback for overall job behaviors, goals and development.

There is a way around the reporting limitation. Because the program's database is managed in Microsoft Access, the user can create custom reports by exporting the data and using Access's report-writing function.

CompStar does a good.job of tracking respondents but does not provide for notifying them of pending evaluations by e-mail. Also, the program's respondent categories are limited to "self," "supervisor" or "peer." Other respondents, such as "customers" and "direct reports," must be placed in the "peer" category.

"Limiting respondent categories was not a problem for our project at Sharp Electronics," says Susan C. White, an external implementation consultant from HR Performance Solutions in Wayne, N.J. "Sharp's application is for high-performance technical teams, so we are primarily concerned with peer feedback Initial setup of the system was more difficult than expected, but once complete, it really hummed- and employees found it very easy to use. We like it because it offers so much flexibility for the price. However, we'd prefer to see the reporting function improved and discovered that the rating-level descriptions did not show up on the data entry screens. This required us to provide employees with an explanation of the ratings before they could enter their evaluations."

20/20 Insight GOLD and Intelligent Consensus could perform all desired functions and gave greater flexibility in reporting and respondent options. They also have many more media distribution options, such as paper forms, diskettes and e-mail attachments through the Internet and intranets. Both systems have integrated a respondent notification feature to alert respondents of pending evaluations. But the additional features are probably overkill in this instance - and carry a higher price tag. CorporatePulse/PulseTools (CorporatePulse) could meet the price demands and deliver equally well on all criteria except one. It does not track individual respondents because it is designed to maintain ultimate confidentiality.

Going beyond the basics. In the initial stages of 360 [degrees] feedback, you may need help orienting your employees to the process and assistance in selecting and writing valid questions. You want an efficient highly flexible, confidential system with many reporting options and graphic displays. You want built-in action planning, development options and suggested resources.

Three of the systems will meet most of those needs at varying levels. All four offer considerable flexibility, customization and resources for questionnaire development and reporting functions. CorporatePulse provides three days of extensive training. While participants spend part of their training on system administration, they focus primarily on writing good questionnaire items and procedures for pilot testing the questionnaires. During the last half-day of training, participants work on a "live" project and receive individual assistance with questionnaire design. The software also contains an extensive library of pretested questions. Both the training and the library items are included in the base price.

"We found the training CorporatePulse provided was excellent. It helped us customize our questions and fie them directly to our leadership academy," says Sherri Sayre, a member of the leader enhancement and associate development team for NationsBank in Jacksonville, Fla. "We also needed media options. Not everyone in the company has computer access or is comfortable using a computer. Generating scannable forms directly from the system was a real plus for us."

20/20 Insight GOLD also includes a broad library of well-researched questions in its base price. However, users incur a separate charge for one-and-a-half days of systems-focused training for administrators. By contrast, Intelligent Consensus has an extra charge for accessing the company's survey library, but the base price includes two days of system administrators' training.

SpeciFic planning and implementation issues are extensively addressed by 20/20 Insight GOLD's bonus module, The 360 Smart Kit. It includes information on the 360 [degrees] process, suggested PowerPoint presentation slides for explaining the project to employees and management and lesson plans for teaching employees how to provide appropriate performance feedback.

The Project Planner, a module of CorporatePulse, assists with project administration. The planner provides a time-line tracking function of all project steps involved in general survey development, administration and analysis. The program's online tutorial gives additional - but limited - help with project planning and communication strategies for 360 [degrees] feedback.