HFC Registration FAQ

FAQ about the Hybrid Forecasting Competition (HFC) Program and the HFC Challenge are below. If you still have questions about HFC, you can email us at [email protected].

WHAT IS THE HFC PROGRAM?

Sponsored by the Intelligence Advanced Research Projects Activity (IARPA), HFC is a research and development program aimed at improving geopolitical & geoeconomic forecasting by combining elements of human- and machine-driven forecasting systems. Human forecasting systems include “crowd wisdom” techniques like prediction markets and probability surveys. Machine systems include statistical models and automated text analysis tools, among many others. HFC will develop and test methods to optimize the combination of human and machine analysis to produce maximally accurate forecasts. Methods developed in HFC may include but are not limited to (1) protocols that train human forecasters to optimally combine human and machine forecast inputs; (2) new predictive models that incorporate both machine data and human judgments; and (3) algorithmic forecasting agents or aids that interact with human analysts.

WHAT IS FORECASTING?

Different people will offer varying definitions of “forecasting.” We define it as the assignment of probabilities (from 0 to 100%) to the outcomes of real-world questions regarding future events (e.g., Will Leader X remain in power through the end of the year?). All HFC forecasting systems will produce probability forecasts against questions like this. As a research participant, you will engage in analytic activities to support the generation of these forecasts. Read on to learn more.

WHAT IS IARPA?

The Intelligence Advanced Research Projects Activity (IARPA) invests in high-risk, high-payoff research programs to tackle some of the most difficult challenges facing the Intelligence Community.

WHY “HYBRID” FORECASTING METHODS?

Intelligence analysts, researchers, and other professionals often wrestle with the problem of how to compare, weight, or otherwise integrate human judgments and machine-based forecasts (e.g., statistical models) to produce an overall forecast. “Crowd wisdom” methods (like prediction markets or probability surveys) that draw heavily on human judgments can be quite accurate and can be applied to a broad range of questions, including very specific and novel topics that are just beginning to make headlines. Machine methods, such as statistical models or automated text analysis tools, can analyze massive quantities of information more rapidly and predictably than humans, and machine methods also frequently display high levels of predictive accuracy. HFC will allow us to better understand how to harness and combine the best of human-intensive and machine-intensive forecasting methods.

WHEN DOES HFC BEGIN?

HFC is a multi-phase program, but your participation can begin now (see below for next steps). If you join HFC as a research participant, you will be randomly assigned to a HFC research system that will compete in a forecasting Tournament to see which system can produce the most accurate forecasts. The formal Tournament will commence in early 2018. In the meantime, starting August 2017 you will be able to hone your forecasting skills by participating in the “HFC Challenge” hosted on the Good Judgment Open (GJO) forecasting site. The HFC Challenge will allow you to explore forecasting procedures, forecast against real-world questions, and check your aptitude for geopolitical & geoeconomic forecasting. While optional, participation in the HFC Challenge is an ideal way to prepare for the Tournament.

WHAT WILL I DO AS AN HFC RESEARCH PARTICIPANT?

Your first steps are to sign up, review and agree to the terms of participation, and complete a registration intake survey. After that, you will be invited to practice your applied forecasting skills as part of an “HFC Challenge” hosted on the Good Judgment Open (GJO) public forecasting platform. The HFC Challenge will start in August 2017 and end sometime in early 2018. Shortly after the HFC Challenge wraps up, you will be randomly assigned to a prototype hybrid forecasting system where you will be part of a multi-month, multi-team hybrid forecasting Tournament. The first season of the Tournament is expected to last approximately seven months. During the Tournament, you will be asked to engage in forecasting activities best suited to optimize the hybrid system to which you are assigned. For example, some hybrid systems may ask you to continue making probability forecasts like you did on the HFC Challenge, others may ask you to gather and evaluate geopolitically & geoeconomically relevant news texts, others may ask you to evaluate statistical models or data sources. These are just a few examples. You will find out more on the particulars of your system when the Tournament begins in early 2018.

WHAT KINDS OF REAL-WORLD FORECASTING QUESTIONS WILL I WORK ON IN HFC?

You can expect to see questions about global politics, diplomacy, military issues, unrest, economics and finance, infectious diseases, and other topics. Questions will typically ask whether some event will occur by some stated deadline or will ask about future quantities (e.g., number of disease cases, economic indicator values).

WHAT’S THE DIFFERENCE BETWEEN THE HFC TOURNAMENT AND THE HFC CHALLENGE?

The Challenge comes first (starting August 2017) and will span approximately 5-6 months. The Tournament comes second (starting early 2018) and the first forecasting season will last approximately 7 months. The Challenge is optional and is intended to prepare you for the Tournament. Once the first season of the Tournament concludes, we expect subsequent Tournament seasons, approximately one per year for potentially four years. The Challenge is designed to provide you with experience making probability forecasts against real-world geopolitical & geoeconomic forecasting questions. This will help you prepare for the Tournament. After the Challenge ends and before the Tournament begins, you will be randomly assigned to one of competing hybrid systems. During the Tournament, these systems will compete to see which can generate the most accurate overall forecasts. Your specific role and forecast-related activities during the Tournament will depend on how that unique hybrid system works: different hybrid approaches are expected to use human input in different ways.

WHEN DO THINGS START AND END?

The HFC Program spans approximately four years, starting in August 2017 with the pre-Tournament HFC Challenge. The Program will be divided into phases (forecasting seasons) to allow forecasters periodic downtime, and to enable performers to assess their results and tweak their systems for optimal forecasting accuracy. The HFC Challenge, spanning approximately five months, commences August 2017 and concludes near the beginning of 2018. The first forecasting season of the Tournament begins shortly thereafter and will last approximately seven months. Additional tournament phases/seasons are expected to follow.

HOW DO I QUALIFY TO PARTICIPATE IN THE HFC PROGRAM?

During the registration process you will need to review and agree to the terms of participation and complete an intake survey. Then you can take part in the pre-Tournament HFC Challenge (starting August 2017), where you can practice making probability forecasts for questions like those that will appear during the Tournament that starts in early 2018. Though participation in the HFC Challenge is not mandatory, those who partake in the Challenge will be in the first groups of forecasters assigned to hybrid forecasting systems for the formal Tournament.

WHAT KIND OF PARTICIPANTS ARE YOU LOOKING FOR?

We are looking for participants who have a keen interest in global events, the drive to learn more, and a willingness to actively forecast throughout the program. There are no other special qualifications or prerequisites to forecast in HFC.

HOW MUCH WORK IS IT? WHAT’S THE TIME COMMITMENT?

Once you are assigned to an HFC system and the Tournament begins (early 2018), forecasters are expected to spend a minimum of two hours per week engaging in forecasting-related activities on their assigned system (a website). Though not required, extra effort beyond two hours will be very beneficial to the research and will help you to further hone your skills. Research shows that more forecasting activity leads to increased forecasting accuracy.

HOW LONG DO I HAVE TO COMMIT AS A PARTICIPANT?

Ideally, a forecaster will participate throughout the duration of the program (expected to be approximately four years), but this is not a requirement. At a minimum, we’d like participants to successfully complete at least one phase (season) of the program. Each phase will last approx. 7-9 months. We understand events can occur that may require a forecaster to withdraw or pause their participation. Your participation is voluntary, and you may withdraw at any time.

HOW IS HFC A COMPETITION? WHO IS COMPETING?

HFC is a competition between different forecasting methods, including different approaches to hybrid (human+machine) forecasting. So, one form of competition will occur at the overall hybrid system level: The hybrid system to which you are assigned will be competing against others. However, there are other forms of competition. Most systems will set analytic goals or tasks for you to complete, and you may be engaging in healthy competition with others working on the same system. In some cases, you may be assigned to a team within your system, where your team will compete with other teams. This will vary depending on the system to which you are assigned. Ultimately, you’re competing with yourself: Trying to get increasingly more accurate and effective in your forecasting performance.

WHAT DO YOU MEAN BY AN HFC TOURNAMENT “SYSTEM”?

A system is simply an interactive website to which you are assigned where you will log on and engage in forecast-related analytic activities. Such activities may include information gathering, comparing different types of information, dialoguing with other participants, and formulating judgments that will help your HFC system produce its overall forecasts. The specifics will depend on the Tournament system to which you are assigned. For the pre-Tournament HFC Challenge, the “system” will be the Good Judgment Open, a public forecasting platform where you will make probability judgments about events and optionally participate in comment/discussion threads.

CAN I CHOOSE THE HYBRID TOURNAMENT SYSTEM TO WHICH I’M ASSIGNED?

No. An essential element of HFC Tournament research is that participants are randomly assigned to hybrid systems. This ensures that all systems start off on equal footing in terms of the number and types of people who contribute to each system’s forecasts.

WHO WILL MANAGE MY TOURNAMENT SYSTEM?

Most participants will be randomly assigned to a forecasting system designed, hosted, and managed by an independent research organization funded by IARPA but not directly affiliated with the HFC Evaluation Team. This is the goal of the HFC Program: for a set of independent research organizations to test their hybrid systems in competition with the others. Although these independent research organizations are also funded by IARPA, each organization will independently design and oversee its system. The HFC Program will evaluate those systems, and your participation is vital to that evaluation.

IS PARTICIPATION IN HFC ANONYMOUS OR CONFIDENTIAL?

As part of recruitment, we request that you provide your name and a persistent email address. We will share the email address you provide with the managers of the HFC Tournament system to which you are assigned. We will instruct them not to disclose your email address. Nevertheless, data leaks can still happen, and participating in HFC means accepting some risk that the data you share could eventually be publicly linked to your email address or name.

WHAT HAPPENS WITH THE INFORMATION I SHARE?

As part of this registration process:

This issue is covered in greater detail within our terms of participation.

WHO CAN SEE MY FORECAST DATA?

In principle, you should assume that anyone can see your forecast data (e.g., probabilities, option choices, unstructured comments). Data on the HFC Challenge is publicly viewable by anyone, and data you produce on your Tournament system will be readily available to the researchers managing that system, your fellow research participants (in most cases), and to the HFC Evaluation Team. If you are concerned about your forecast data being associated with your identity, you are encouraged to select a user/screen name that does not directly identify you. More details on data sharing and confidentiality can be found on our terms of participation form.