CorpComms Magazine

It was the word 'wonderful' that confused the computer. Despite the widespread condemnation of the banking crisis that rippled across Britain's financial system, and public anger at the hugely expensive government bail-outs of Northern Rock, Royal Bank of Scotland, HBOS and new parent Lloyds, computerised evaluation of the coverage suggested a positive tone.

'Wonderful kept coming up, but it was being used in a negative way,' says Paul Hender, media analysis director at Metrica. 'The coverage was actually sarcastic. People were saying How wonderful it is that our taxes are being used to bail out the banks. Sarcasm is still a problem for computers. They are not going to get all the nuances.'

Similar problems emerge when a word with a negative connotation, such as cancer or virus, is used in a positive manner. 'We did work for Cancer Research,' adds Hender. 'About 90 per cent of its coverage included the word 'cancer' but in a positive way that recognised the role of the charity.' Automated analysis might have viewed this coverage as negative, although it is possible to 'retrain' the sentiment model on more relevant coverage.

Other difficulties arise during the evaluation of text language, where abbreviations are often used, and which can be used in the comments appended to social media, such as blogs.

It is problems such as these that are held up by some market participants as evidence that the personal touch is the only solution for media evaluation. They dismiss the counter-arguments by automated analysis supporters, who claim that personal evaluation can be affected by whether individual has had a bad journey to work or an argument with their partner.

Katie Paine, an American-based evaluation expert who was founding member of the International Public Relations' Commission on Research and Evaluation, has commented that while computers have learned to understand words, they still do not understand nuances of conversations, adding: 'Facebook, HP and Microsoft did extensive research before selecting measurement tools, and all three insisted on human analysts. So I ask you if some of the leading players in technology don't trust computers, why should you?'

Marcus Gault, managing director at Precise Media, believes that the only reason to justify choosing automated evaluation services is cost. Computer based analysis might cost around five pence an article while a personal evaluation could cost between £3 and £5 per article. 'It is much cheaper because the companies do not employ vast swathes of analysts,' he says. 'But the problem is that it just doesn't work yet.'

LIMITATIONS

John Gaylor, digital products manager at Durrants, agrees, adding: 'There is a limit to what a machine can do.' He believes that media evaluation is a bespoke product that requires careful interaction between analysts and customers to ensure deep understanding of the brief.

'Our analysts will have an understanding of a client's key messages and recognise if these are coming through,' says Gaylor. 'They can also identify new messages as they come through. It is about being proactive. A machine cannot pick those sorts of things up. It can't be proactive.'

However, some evaluation experts believe there is a role for automated analysis to play. Jerry Ward, managing director of Press Data, believes it can be used to evaluate vast swathes of media and produce impressive reports for the board.

'Very few professional PR people would find this of value or think This is brilliant but it is quite good to show the board. It doesn't really qualify or assess anything but it shows outputs. For communications professionals on performance related pay, the reports demonstrate that they are doing something. It is a report filled with pretty pictures,' he says.

'However, in many ways this is no more valid an evaluation than the old days of dropping a large pile of press cuttings onto the desk of the PR manager and using the loudness of the thud to quantify the success of the PR.'

Ben Lloyd, sector director at Echo, which recently launched a low cost automated evaluation system, EchoSonar, for clients, believes there are other more tangible benefits.

He says: 'Automated analysis based on computer programming can provide quick and straightforward metrics to show how well a company name, brand, issue or competitor has been picked up. More complex systems allow you to track messages too, but given this is based on technology 'reading' the article, you are to some extent reliant on what keywords are included in the search.'

EchoSonar allows clients to get an overview of their online media profile at the touch of a button, and also to keep an eye on competitors. There are also links to the original articles. Lloyd believes the product works 'incredibly well for clients who want to keep up to date with their competitor media profiles on a daily basis rather than have the full detail of analysis'.

ROLE BUT NOT REPLACEMENT

Other advocates of the personal agree believe that there may be a role for automated evaluation to play, but discount suggestions that it should replace living analysts.

With tens of thousands of articles appearing online every day, Metrica's Hender believes that there is a role for automated analysis as a support function. He believes organisations should use personal evaluation for traditional media outlets that are of particular importance, for example the Financial Times for listed companies. 'But if they want to know what how their business is perceived in social media, then there is a strong argument for automation.' He also points out that a computer can trawl through hundreds of thousands of articles and identify occasions when a company gets a name check. A human analyst would almost certainly miss some appearances, although the number of articles checked would also be reduced.

'What automated media evaluation can give is quick, almost real time, stats. There is a place for both types of services,' says Press Data's Ward. 'But it is important to understand how to use each. Any experienced PR operator will understand the difference. If they believe that automated evaluation will provide a real impression of how successful they have been, they are either deceiving themselves or their organisation, and missing the opportunity to gain meaningful insight into how they can build successful communications.'

Gault concludes: 'I would love to get software to do evaluation. It would help my pricing and profits. It is fine for word and sentence matching but what clients really want is sentiment, implied comment and to understand attributes associated with the organisation. We test automated evaluation software every year, and the problem is that the ones in the market are just not fit for purpose. That is not to say that they may not work in the future, but they're not there now.' 

share me: del.icio.us | digg | reddit | Tweet