New Software Aids in Assessment

Facing greater demands for accountability, colleges turn to technology to
analyze mounds of data

By DAN CARNEVALE

http://chronicle.com/weekly/v53/i30/30a03701.htm

From the issue dated March 30, 2007

Richmond, Va.

The last time Virginia Commonwealth University had to prepare for an accreditation review, officials here found themselves overwhelmed with data.

The university's accreditor, the Southern Association of Colleges and Schools, was asking for more information than ever before about how much students were learning: grades, test scores, written evaluations, and other measures.

Much of that information was scattered throughout the institution — kept in computer files and storage drawers. So Jean M. Yerian, then the director of assessment, led the development of a computer program that would organize and analyze all the assessments of students being done on the campus.

The computer program, dubbed Weave, not only helped the university satisfy its accreditors, but also appealed to other colleges, which wanted to use it to prepare for their own accreditation reviews.

"We started out as solving our own problem and ended up developing something that can help others as well," says Ms. Yerian.

Last year Virginia Commonwealth spun off the project as an independent company called WeaveOnline. Ms. Yerian resigned her post at the university last month to become director of assessment management for the company, which has already attracted more than 40 colleges as clients.

Other colleges, facing increasing pressure from accreditors, state lawmakers, and the federal government to prove that their students are actually learning something, are looking for similar help. College officials say they know they can meet expectations, but proving it to regulators is going to take some extra work and some technology. They are seeking computer programs to manage and analyze the gargantuan amounts of assessment data.

Supply is slowly meeting the demand. Companies such as Blackboard, Desire2Learn, and Datatel have developed software that helps conduct institutional assessments. Other companies, such as Oracle and eCollege, have plans to jump into the game as well.

But such developments have made some faculty members uneasy. The same tools that give extensive data to accreditors could also let administrators pry into academics' lives in the classroom and micromanage their work.

Burden or Benefit?
While aware of those concerns, college officials say they recognize the writing on the wall. With tuition rising nationwide and increased competition from foreign universities, public pressure is mounting for colleges and universities to prove their worth. Secretary of Education Margaret Spellings is looking for ways to make institutions more accountable.

Among the ideas that have come out of her Commission on the Future of Higher Education are the creation of a "unit record" database to track students' progress through college and additional accreditation standards measuring what students learn.

To head off heavy regulation from the federal government, colleges are working with accreditors and private companies to develop methods and technology to assess all levels of institutional achievement.

Ana B. Borray, director of marketing for executive programs at Datatel, says regulators want colleges to use the assessment data bases for more than bean counting. Institutions should show they are actively using the data to improve programs and help students who may be experiencing problems.

For example, legislators are demanding that colleges improve graduation rates. If colleges find that students are taking too long to finish their degrees, she says, they can use the assessment data to find ways to help struggling students before they get too far behind.

"It's not only about reporting the outcomes," Ms. Borray says. "It's about using software to help you change the outcomes."

Paper Overload
Like many institutions, Virginia Commonwealth had already been doing its own education assessments for years. In addition to keeping track of student grades and test scores, professors recorded evaluations of general areas such as student writing abilities and critical-thinking skills. And specific departments used their own assessments, such as evaluations of mechanical-engineering skills or understanding of biology.

But most of the results were kept on paper. Binders literally took up entire rooms on the campus. That made it difficult to spot trends and to compile the results for outside evaluations.

The university created Weave so that as professors performed the assessments on their students, they could record the information in the central database. Deans and other administrators would set goals for reaching academic objectives and study the results.

When the data are compiled, administrators are better able to see which goals have been met and which need work. They can use the same system to create methods for improvement. The stystem also provides quantifiable proof to regulators about how a program is performing.

The university has already seen positive results from the use of Weave, says Laura J. Moriarty, vice provost for academic and faculty affairs at Virginia Commonwealth. For example, many students could not discern the quality of various sources for research, she says. Using Weave helped faculty members and administrators recognize that this was a problem throughout the university. The software also allowed them to measure whether students were making improvements.

"We discussed this as a faculty," she says, "and decided that all students needed a refresher in all the lower-level courses about using primary sources and about using more academic sources instead of using Web sites and electronic encyclopedia entries."

Weave also measures success in nonacademic areas, such as internship programs, student job placements, and faculty activities. The quality of back-room functions, such as financial aid and public relations, can also be determined. "These are things that may not have anything to do with a course, but it is very important for the program," says Barton B. Cregger, associate dean of the School of Engineering.

Big Brother's Watching
To get a complete picture of the university, Virginia Commonwealth administrators require that all faculty and staff members use Weave. But other institutions that use the system have decided to make it optional, for fear of upsetting faculty members.

Jonathan Schnyer, associate director of institutional assessment and studies at the University of Virginia, says faculty members do not like to be forced to do anything, especially when it involves administrators looking over their shoulders.

"It's an assessment coordinator's dream and a faculty member's nightmare," he says of the software. "It gives you the ability to see what people are doing and whether they're doing what they're supposed to."

He says the system covers a representative sample of all 257 academic programs at the university, but he estimates that fewer than half of faculty members use it. So administrators must extrapolate from the volunteers who do participate.

"It's a little messy, but I think it's a far more effective approach," Mr. Schnyer says. "We're not Big Brother. We're here to assist."

'Broader Impact'
The University of Virginia was Weave's sixth client, Mr. Schnyer says. When the institution was up for reaccreditation, officials realized the focus was shifting toward assessment of student learning and other factors. "You have to prove to them that you're not only doing it but that you're using the results," Mr. Schnyer says. "That was the impetus for us."

The program has worked well for the university, he says. Officials get a better snapshot of what is going on at all levels of education.

The University of Virginia paid Virginia Commonwealth $33,000 for the initial setup costs in 2006 and pays $10,000 a year to maintain the Web-based service, Mr. Schnyer says. He calls the price a "bargain."

Blackboard released its own assessment software, called the Blackboard Outcomes System, in January. According to David A. Yaskin, the company's vice president of product marketing, it has attracted several buyers in the past few months, including institutions that helped develop the software. The new system is separate from the company's other software but is designed to work hand in hand with it.

Mr. Yaskin says Blackboard simplifies the assessment process by linking to the company's course-management software, which is in most clients' classrooms already. Much of the work that students do is automatically uploaded into the database. "You can literally grab assignments from students," he says.

Paul E. Fisher, director of the Teaching, Learning, and Technology Center at Seton Hall University, helped Blackboard develop its assessment software over the last two years. The university has tested the software and now plans to start using it.

The software focuses its assessment at the classroom level, measuring students' specific skills, such as oral and written communication, and their electronic portfolios, or online collections of their work.

Building on those measurements, administrators can evaluate whole programs. "It's really a matter of implementation," Mr. Fisher says. "We can get a real picture of what the institution is doing."

With large companies like Blackboard and Oracle moving into this market, smaller businesses like WeaveOnline may face some tough competition. But officials say they are not worried.

Ivelina S. Metcheva, director of technology transfer for Virginia Commonwealth, says the growing competition has actually helped WeaveOnline in the short term because it has brought more attention to the need for these kinds of assessment tools.

"Competition is good because it validates the market," Ms. Metcheva says. "There is a need for this kind of product."

http://chronicle.com

Section: Information Technology
Volume 53, Issue 30, Page A37

Copyright © 2006 by The Chronicle of Higher Education

Subscribe | About The Chronicle | Contact us | Terms of use | Privacy policy | Help