OONI Probe ASN Incident Report
Last week we noticed
that some OONI measurements were available on OONI Explorer under a report ID containing a
valid Autonomous System Number (ASN), even though the raw
JSON data contained zero as the ASN (i.e. encoded as
initially thought that this was caused by a bug in our API code, but it
actually turned out to be an OONI Probe
bug in our probe engine (which
powers the OONI Probe apps).
Please update your OONI Probe mobile app to the latest version (2.7.0), which fixes this bug.
If you are an OONI Probe desktop app user and prefer not to share your
network ASN, please refrain from running tests until we have released a
fix for OONI Probe desktop (hopefully this will be possible next week).
When we investigated this issue, we
found that the probe
was opening a report with the real ASN of the user’s network, regardless
of whether the user had opted out of sharing their ASN with OONI.
Because the OONI backend uses the ASN submitted by the probe to
construct the report ID, the actual ASN was leaked in the report ID of
In practice, this means that if you disabled the “Include Network Info”
setting in your OONI Probe app — which would normally disable the
collection and publication of your network ASN (e.g.
“Vodafone Italia”) — your network ASN was not published in the OONI
Explorer measurement page or in the raw JSON data (where it was
AS0), but it was included in the report ID of those
As soon as we noticed this bug, we swiftly worked on resolving this
incident. Specifically, we:
Audited the probe-engine codebase to determine
whether there were other similar issues. We found that the user
network ASN (when the user had opted out of ASN collection) is not
only leaked in the report ID, but that in some cases, the network
name (such as “Vodafone Italia”) is displayed in the raw data,
even though the annotated ASN is
AS0. We also realized that
because we don’t sanitize the resolver IP (as this information is
useful for measuring DNS consistency), it has been possible to
obtain the ASN through it (ever since the OONI Web Connectivity test was released in
to our Go probe engine that fix this bug and sanitize the resolver
IP. These changes not only ensure that the user network ASN is
no longer leaked through the report ID of measurements (when users
have opted out of ASN collection), but they also help ensure that
the network name is not disclosed when measurements are annotated
AS0, and that the ASN is not retrieved through the
resolver IP. We also implemented tests to ensure that the fixes
work as expected.
Released a new OONI Probe mobile app version (2.7.0). We fixed the bug on
OONI Probe Android
and released OONI Probe Mobile 2.7.0 with the bug fix.
Fixed the bug on OONI Probe desktop. We fixed the bug in the OONI
Probe desktop app, but unfortunately it has not been possible to
make a release yet (as third party action is required). We are
hoping that it will be possible to release the OONI Probe desktop
app with the bug fix next week.
Evaluated how many global OONI measurements were affected by this bug. According to our analysis, only around 2% of global OONI
measurements leaked the user network ASN in the report ID (this
mainly involves new probes), and about 12% of global OONI
measurements might have disclosed the ASN through the client
resolver in OONI’s Web Connectivity test (this mainly
involves legacy probes).
Made changes to OONI Explorer to make
AS0 measurements harder to find. As
a short-term solution, we removed the listing of measurements
AS0 (when users have opted out of ASN collection)
from the OONI Explorer Search page, and we ensured that
users cannot filter OONI Explorer measurements based on
For extra precaution, we also made OONI Explorer measurements
AS0 return a 404 error, instead of the
measurement page and raw data.
As a next step, we plan
to remove past
AS0 measurements from the OONI API entirely. We will also explore whether we
can make changes to the fast-path pipeline so that all past
measurements are sanitized. But this is a very complex task, and we
unfortunately are unable to make these changes at the moment, as we are
busy migrating our infrastructure over to another data centre and
building a new PostgresDB to accommodate this migration.
We will also evaluate whether we can release a fix for the legacy version of OONI Probe (which is
primarily run on Linux platforms). However, according to our analysis,
most legacy users are using ooniprobe 2.1.0 as packaged by Debian
jessie. But because no subsequent OONI Probe versions have been packaged
for Debian, it seems that tagging a new release of the legacy probe is
not going to solve the original problem. Since that version of OONI
Probe is deprecated, we recommend the use of the OONI Probe Command Line Interface (CLI) instead
(version 3.0.8 of probe-cli
contains a fix for this issue). Over the next year, we aim to release
Linux packages for OONI Probe which would serve as a replacement for the
legacy version of OONI Probe.
We decided to publish this incident report at this time because the
issue is solved on OONI Probe mobile platforms (which account for the
majority of global OONI Probe users) and on OONI Explorer. We are currently unable to
release the OONI Probe desktop app with the bug fix because we are
waiting on a third party for a signing certificate. We nonetheless chose
to inform our community in advance and offer a solution in the interim
(update to the latest OONI Probe 2.7.0 mobile app version and stop running OONI Probe
desktop tests if you are concerned about disclosing your network ASN).
Most OONI Probe users were not affected by this bug, since roughly
86% of OONI measurements collected from around the world did not
disable the collection and publication of network information, which is
enabled in the default settings.
The reason why we have this setting enabled by default is because
identifying the network on which tests are performed is important in
evaluating where potential internet censorship took place. Without this
information, if a measurement shows the blocking of a website, we can
know that this blocking took place in a given country (because OONI
Probe users cannot opt out of country
code collection), but we don’t know on which network the blocking
Knowing the network on which internet censorship is implemented is
important because it:
Increases transparency of internet censorship implemented by
different Internet Service Providers (ISPs) in a country (and
allows the public to evaluate whether that censorship is in
compliance with the country’s laws and government orders);
Allows us to potentially compare internet censorship across ISPs in
a country (particularly since we often see that internet
censorship varies across networks);
Allows us to track censorship changes over time on a network;
Improves OONI data analysis capabilities and helps with confirming
censorship events with more confidence (since it is important to
look at aggregate measurements on a specific network over time).
Due to the above reasons, given the very limited value of measurements
AS0, and in light of the fact that a small percentage
of global OONI measurements opt out of ASN collection, we are
considering the possibility of removing this option (i.e. to opt
out of ASN collection) entirely from the OONI Probe app settings. For
the time being though, we have fixed this bug
and released a fix for our users.
We have always aimed to provide OONI Probe users with as much choice as
possible (in terms of data collection settings), because we recognize
that people run OONI Probe in different environments, have different
threat models, and feel comfortable with different options.
Since 2012, we have gone to extra lengths in order to provide our
community with as much choice as possible in terms of data collection
options. But we have learned (from this and from a previous incident) that
this adds extra complexity to our codebase, making the detection and
prevention of such bugs harder. This is particularly the case given the
many large and complex codebases that we
maintain, and our limited resources (we’re a small team in the nonprofit
The OONI team sincerely apologizes to the OONI community for this
incident. We would never intentionally harm our users, we value and
respect user choice, and we take seriously the trust our users have
placed in us. We do our best to give you as much control over how you
use OONI Probe, but sometimes we make mistakes.
To limit the possibility of such bugs recurring in the future, we are
working on improving our development practices. To this end, we will
Proactively search for sources of complexity and remove them with
the goal of reducing the overall complexity of our codebases;
Improve how we do testing of our apps (both in terms of unit
tests, but also end-to-end integration tests);
Strengthen our quality assurance and release management
Be more thorough and rigorous in how we do code review for all
Ensure that OONI developers follow the best practices outlined in
our software development guidelines.
To learn more about our data practices and about the principles that
govern OONI data collection, please refer to our Data Policy.
If you have any questions related to this incident or anything else OONI
related, please do not hesitate to reach out to us at: