Australian Security and Intelligence Conference
Document Type
Conference Proceeding
Publisher
SRI Security Research Institute, Edith Cowan University, Perth, Western Australia
Abstract
In 2010, the US Intelligence Advanced Research Projects Activity (IARPA) announced a 4-year forecasting “tournament”. Five collaborative research teams are attempting to outperform a baseline opinion pool in predicting hundreds of geopolitical, economic and military events. We are contributing to one of these teams by eliciting forecasts from Delphi-style groups in the US and Australia. We elicit probabilities of outcomes for 3-5 monthly questions, such as: Will Australia formally transfer uranium to India by 1 June 2012? Participants submit probabilities in a 3-step interval format, view those of others in their group, share, rate and discuss information, and then make a second private judgement. Performance is assessed using Brier scores. After Year 1, we ranked second of five teams in the competition. The Brier scores from the US Delphi groups improved on the baseline scores by 10%, the prediction market operated by our team in the US beat the baseline by 47%, and the Australian Delphi groups outperformed the baseline by 51% (answering different, matched questions to the US groups). The Australian groups were more socially and demographically diverse than the US groups. Group diversity may be an important factor determining the forecasting performance of the aggregated predictions.
DOI
10.4225/75/57a03816ac5cf
Comments
5th Australian Security and Intelligence Conference, Novotel Langley Hotel, Perth, Western Australia, 3rd-5th December, 2012