HFC Registration FAQ

FAQ about the Hybrid Forecasting Competition (HFC) Program and the HFC Challenge are below. If you still have questions about HFC, you can email us at [email protected].

WHAT IS THE HFC PROGRAM?

Sponsored by the Intelligence Advanced Research Projects Activity (IARPA), HFC is a research and development program aimed at improving geopolitical & geoeconomic forecasting by combining elements of human- and machine-driven forecasting systems. Human forecasting systems include “crowd wisdom” techniques like prediction markets and probability surveys. Machine systems include statistical models and automated text analysis tools, among many others. HFC will develop and test methods to optimize the combination of human and machine analysis to produce maximally accurate forecasts. Methods developed in HFC may include but are not limited to (1) protocols that train human forecasters to optimally combine human and machine forecast inputs; (2) new predictive models that incorporate both machine data and human judgments; and (3) algorithmic forecasting agents or aids that interact with human analysts.

WHAT IS FORECASTING?

Different people will offer varying definitions of “forecasting.” We define it as the assignment of probabilities (from 0 to 100%) to the outcomes of real-world questions regarding future events (e.g., Will Leader X remain in power through the end of the year?). All HFC forecasting systems will produce probability forecasts against questions like this. As a research participant, you will engage in analytic activities to support the generation of these forecasts. Read on to learn more.

WHAT IS IARPA?

The Intelligence Advanced Research Projects Activity (IARPA) invests in high-risk, high-payoff research programs to tackle some of the most difficult challenges facing the Intelligence Community.

WHY “HYBRID” FORECASTING METHODS?

Intelligence analysts, researchers, and other professionals often wrestle with the problem of how to compare, weight, or otherwise integrate human judgments and machine-based forecasts (e.g., statistical models) to produce an overall forecast. “Crowd wisdom” methods (like prediction markets or probability surveys) that draw heavily on human judgments can be quite accurate and can be applied to a broad range of questions, including very specific and novel topics that are just beginning to make headlines. Machine methods, such as statistical models or automated text analysis tools, can analyze massive quantities of information more rapidly and predictably than humans, and machine methods also frequently display high levels of predictive accuracy. HFC will allow us to better understand how to harness and combine the best of human-intensive and machine-intensive forecasting methods.

WHEN DID HFC BEGIN?

HFC officially began March 7th, 2018. HFC is a multi-phase program (see below for next steps). You can reserve your spot now for the second season of the competition, beginning in Fall 2018.

WHAT WILL I DO AS AN HFC RESEARCH PARTICIPANT?

You will be randomly assigned to a prototype hybrid forecasting system in Fall 2018 where you will be part of a multi-month, multi-team hybrid forecasting Tournament. The first season of the Tournament runs March 7 through September 7th 2018. During the second season of the Tournament, you will be asked to engage in forecasting activities best suited to optimize the hybrid system to which you are assigned. For example, some hybrid systems may ask you to continue making probability forecasts like you did on the HFC Challenge, others may ask you to gather and evaluate geopolitically & geoeconomically relevant news texts, others may ask you to evaluate statistical models or data sources. These are just a few examples. You will find out more on the particulars of your system when the second season of the Tournament begins in Fall 2018.

WHAT KINDS OF REAL-WORLD FORECASTING QUESTIONS WILL I WORK ON IN HFC?

You can expect to see questions about global politics, diplomacy, military issues, unrest, economics and finance, infectious diseases, and other topics. Questions will typically ask whether some event will occur by some stated deadline or will ask about future quantities (e.g., number of disease cases, economic indicator values).

WHEN DO THINGS START AND END?

The HFC Program spans approximately four years. The Program is divided into phases (forecasting seasons) to allow forecasters periodic downtime, and to enable performers to assess their results and tweak their systems for optimal forecasting accuracy. The first forecasting season of the Tournament began March 7th 2018. The next season is expected to start in Fall 2018 and will last approximately nine months.

HOW DO I QUALIFY TO PARTICIPATE IN THE HFC PROGRAM?

During the registration process you will need to review and agree to the terms of participation and complete an intake survey. The first phase of the Tournament began in March 2018 and is currently closed to registration, but the second phase tentatively begins in Fall 2018. You will be contacted with further participation details prior to commencement of the second phase.

WHAT KIND OF PARTICIPANTS ARE YOU LOOKING FOR?

We are looking for participants who have a keen interest in global events, the drive to learn more, and a willingness to actively forecast throughout the program. There are no other special qualifications or prerequisites to forecast in HFC.

HOW MUCH WORK IS IT? WHAT’S THE TIME COMMITMENT?

Forecasters are expected to spend a minimum of two hours per week engaging in forecasting-related activities on their assigned system (a website). Though not required, extra effort beyond two hours will be very beneficial to the research and will help you to further hone your skills. Research shows that more forecasting activity leads to increased forecasting accuracy.

HOW LONG DO I HAVE TO COMMIT AS A PARTICIPANT?

Ideally, a forecaster will participate throughout the duration of the program (expected to be approximately four years), but this is not a requirement. At a minimum, we’d like participants to successfully complete at least one phase (season) of the program. Each phase will last approx. 7-9 months. We understand events can occur that may require a forecaster to withdraw or pause their participation. Your participation is voluntary, and you may withdraw at any time.

HOW IS HFC A COMPETITION? WHO IS COMPETING?

HFC is a competition between different forecasting methods, including different approaches to hybrid (human+machine) forecasting. So, one form of competition will occur at the overall hybrid system level: The hybrid system to which you are assigned will be competing against others. However, there are other forms of competition. Most systems will set analytic goals or tasks for you to complete, and you may be engaging in healthy competition with others working on the same system. In some cases, you may be assigned to a team within your system, where your team will compete with other teams. This will vary depending on the system to which you are assigned. Ultimately, you’re competing with yourself: Trying to get increasingly more accurate and effective in your forecasting performance.

WHAT DO YOU MEAN BY AN HFC TOURNAMENT “SYSTEM”?

A system is simply an interactive website to which you are assigned where you will log on and engage in forecast-related analytic activities. Such activities may include information gathering, comparing different types of information, dialoguing with other participants, and formulating judgments that will help your HFC system produce its overall forecasts. The specifics will depend on the Tournament system to which you are assigned.

CAN I CHOOSE THE HYBRID TOURNAMENT SYSTEM TO WHICH I’M ASSIGNED?

No. An essential element of HFC Tournament research is that participants are randomly assigned to hybrid systems. This ensures that all systems start off on equal footing in terms of the number and types of people who contribute to each system’s forecasts.

WHO WILL MANAGE MY TOURNAMENT SYSTEM?

Most participants will be randomly assigned to a forecasting system designed, hosted, and managed by an independent research organization funded by IARPA but not directly affiliated with the HFC Evaluation Team. This is the goal of the HFC Program: for a set of independent research organizations to test their hybrid systems in competition with the others. Although these independent research organizations are also funded by IARPA, each organization will independently design and oversee its system. The HFC Program will evaluate those systems, and your participation is vital to that evaluation.

IS PARTICIPATION IN HFC ANONYMOUS OR CONFIDENTIAL?

As part of recruitment, we request that you provide your name and a persistent email address. We will share the email address you provide with the managers of the HFC Tournament system to which you are assigned. We will instruct them not to disclose your email address. Nevertheless, data leaks can still happen, and participating in HFC means accepting some risk that the data you share could eventually be publicly linked to your email address or name.

WHAT HAPPENS WITH THE INFORMATION I SHARE?

As part of this registration process:

This issue is covered in greater detail within our terms of participation.

WHO CAN SEE MY FORECAST DATA?

In principle, you should assume that anyone can see your forecast data (e.g., probabilities, option choices, unstructured comments). Data on the HFC Challenge is publicly viewable by anyone, and data you produce on your Tournament system will be readily available to the researchers managing that system, your fellow research participants (in most cases), and to the HFC Evaluation Team. If you are concerned about your forecast data being associated with your identity, you are encouraged to select a user/screen name that does not directly identify you. More details on data sharing and confidentiality can be found on our terms of participation form.