OONI Probe ASN Incident Report

Maria Xynou, Simone Basso 2020-10-09

Last week we noticed that some OONI measurements were available on OONI Explorer under a report ID containing a valid Autonomous System Number (ASN), even though the raw JSON data contained zero as the ASN (i.e. encoded as AS0). We initially thought that this was caused by a bug in our API code, but it actually turned out to be an OONI Probe bug in our probe engine (which powers the OONI Probe apps).

Please update your OONI Probe mobile app to the latest version (2.7.0), which fixes this bug. If you are an OONI Probe desktop app user and prefer not to share your network ASN, please refrain from running tests until we have released a fix for OONI Probe desktop (hopefully this will be possible next week).

When we investigated this issue, we found that the probe was opening a report with the real ASN of the user’s network, regardless of whether the user had opted out of sharing their ASN with OONI. Because the OONI backend uses the ASN submitted by the probe to construct the report ID, the actual ASN was leaked in the report ID of OONI measurements.

In practice, this means that if you disabled the “Include Network Info” setting in your OONI Probe app — which would normally disable the collection and publication of your network ASN (e.g. AS30722 for “Vodafone Italia”) — your network ASN was not published in the OONI Explorer measurement page or in the raw JSON data (where it was displayed as AS0), but it was included in the report ID of those measurements.

As soon as we noticed this bug, we swiftly worked on resolving this incident. Specifically, we:

As a next step, we plan to remove past AS0 measurements from the OONI API entirely. We will also explore whether we can make changes to the fast-path pipeline so that all past AS0 measurements are sanitized. But this is a very complex task, and we unfortunately are unable to make these changes at the moment, as we are busy migrating our infrastructure over to another data centre and building a new PostgresDB to accommodate this migration.

We will also evaluate whether we can release a fix for the legacy version of OONI Probe (which is primarily run on Linux platforms). However, according to our analysis, most legacy users are using ooniprobe 2.1.0 as packaged by Debian jessie. But because no subsequent OONI Probe versions have been packaged for Debian, it seems that tagging a new release of the legacy probe is not going to solve the original problem. Since that version of OONI Probe is deprecated, we recommend the use of the OONI Probe Command Line Interface (CLI) instead (version 3.0.8 of probe-cli contains a fix for this issue). Over the next year, we aim to release Linux packages for OONI Probe which would serve as a replacement for the legacy version of OONI Probe.

We decided to publish this incident report at this time because the issue is solved on OONI Probe mobile platforms (which account for the majority of global OONI Probe users) and on OONI Explorer. We are currently unable to release the OONI Probe desktop app with the bug fix because we are waiting on a third party for a signing certificate. We nonetheless chose to inform our community in advance and offer a solution in the interim (update to the latest OONI Probe 2.7.0 mobile app version and stop running OONI Probe desktop tests if you are concerned about disclosing your network ASN).

Most OONI Probe users were not affected by this bug, since roughly 86% of OONI measurements collected from around the world did not disable the collection and publication of network information, which is enabled in the default settings.

The reason why we have this setting enabled by default is because identifying the network on which tests are performed is important in evaluating where potential internet censorship took place. Without this information, if a measurement shows the blocking of a website, we can know that this blocking took place in a given country (because OONI Probe users cannot opt out of country code collection), but we don’t know on which network the blocking occurred.

Knowing the network on which internet censorship is implemented is important because it:

Due to the above reasons, given the very limited value of measurements annotated as AS0, and in light of the fact that a small percentage of global OONI measurements opt out of ASN collection, we are considering the possibility of removing this option (i.e. to opt out of ASN collection) entirely from the OONI Probe app settings. For the time being though, we have fixed this bug and released a fix for our users.

We have always aimed to provide OONI Probe users with as much choice as possible (in terms of data collection settings), because we recognize that people run OONI Probe in different environments, have different threat models, and feel comfortable with different options.

Since 2012, we have gone to extra lengths in order to provide our community with as much choice as possible in terms of data collection options. But we have learned (from this and from a previous incident) that this adds extra complexity to our codebase, making the detection and prevention of such bugs harder. This is particularly the case given the many large and complex codebases that we maintain, and our limited resources (we’re a small team in the nonprofit world).

The OONI team sincerely apologizes to the OONI community for this incident. We would never intentionally harm our users, we value and respect user choice, and we take seriously the trust our users have placed in us. We do our best to give you as much control over how you use OONI Probe, but sometimes we make mistakes.

To limit the possibility of such bugs recurring in the future, we are working on improving our development practices. To this end, we will continue to:

To learn more about our data practices and about the principles that govern OONI data collection, please refer to our Data Policy.

If you have any questions related to this incident or anything else OONI related, please do not hesitate to reach out to us at: contact@openobservatory.org.