Evaluating OONI’s New Measurement Engine in the Context of the Blocking of Women on Web in Spain
Reproduction rights site www.womenonweb.org has been blocked in Spain over the last months by several local Internet Service Providers (ISPs). Once Spanish hacktivist communities noticed that www.womenonweb.org was inaccessible, they coordinated on running OONI Probe tests to collect measurement data that could serve as evidence. The findings of this investigation have been published in an excellent report, whose writing was coordinated by our former colleague, Vasilis Ververis.
We were involved in the technical review of the report. We found their methodology particularly interesting. They used the OONI Web Connectivity experiment along with OONI’s experimental SNI based blocking methodology, and they also performed manual experiments with OpenSSL and cURL. Because OONI’s mission is to simplify and automate the collection of censorship evidence, we are always interested in learning how we can expand our Web Connectivity methodology to automatically gather more data that is useful in specific cases. Coincidentally, as part of rewriting the Web Connectivity implementation in Go, we have already written experimental code that combines vanilla Web Connectivity with SNI blocking and other techniques. We therefore asked them to run this experimental code with www.womenonweb.org as a target. Our goal was to evaluate whether our working prototype would have allowed them to obtain the same research results, without resorting to manually running specifically tailored network experiments.
This report describes what we learned from this experience. We will start off by describing recent research efforts focusing on SNI based blocking. We will then discuss the blocking of www.womenonweb.org and explain why Ververis et al findings are very interesting in the context of SNI based blocking. Afterwards, we briefly describe our experimental implementation and explain how you can help us improve it. Next, we will describe how we tailored our experimental implementation for the www.womenonweb.org case. We then describe our findings and what we plan on doing next.
SNI Based Blocking Measurements
SNI based filtering appears to have been emerging around the world in recent years. We previously reported on the SNI based filtering of Wikipedia in Venezuela, as well as on the SNI based filtering of Facebook live-streaming in Jordan (among other cases). Very interesting SNI based blocking investigations have also been performed by researchers at India’s Centre for Internet and Society (CIS), who investigated, for example, how the Jio ISP is using SNI based blocking to prevent access to specific websites. Researchers at Jigsaw have also started drafting improved website blocking methodologies that aim at providing the user with specific insights on what blocking techniques are used. The SNI blocking bits of this Jigsaw methodology have already been integrated into OONI’s measurement engine. More recently, we published a research report evaluating our SNI blocking implementation in Iran, where we were able to confirm that our SNI blocking experiment works as intended.
Blocking of Women on Web in Spain
Starting from January 2020, OONI data shows that access to www.womenonweb.org has been blocked on multiple local networks in Spain, as illustrated through the following chart.
This is not the first time that OONI data shows the blocking of www.womenonweb.org. Last year, we reported the blocking of www.womenonweb.org and www.womenonwaves.org in Brazil, Iran, Saudi Arabia, South Korea, and Turkey. More recently, as mentioned above, this problem has been studied and documented by Spanish hacktivists in coordination with our former colleague, Vasilis Ververis (now working on the Magma Project).
They (Ververis et al) subsequently published a report which shares these findings. These are, in our view, the most interesting findings, all occurring on Vodafone Spain:
SNI blocking only happens when connecting to the www.womenonweb.org IP address and, crucially, does not happen when connecting to an unrelated server (which implies that the SNI blocking methodology that we recently implemented would not report any error in this case);
When ignoring the certificate validation error, the returned block page provides further insights that could be useful to characterize the blocking.
We will therefore discuss our experimental implementation architecture, and how we tweaked its implementation to be able to reproduce the three findings above.
Aladdin: our experimental implementation
On May 1st, 2020 we released a new experiment called urlgetter. This experiment is not meant to be directly exposed to OONI Probe users. Rather, we see this as a flexible building block that we will use to implement the improved Web Connectivity experiment.
The gist of this experiment is that you give it a URL to measure. When
https:// URL, this experiment fetches the
related web page. In addition, it also handles custom URL schemes, such
dnslookup://. The former causes the
urlgetter to just perform a TLS handshake; the latter triggers a DNS
Of course, the urlgetter allows us to force a specific SNI. It additionally features a DNS cache, so that the results of a DNS lookup could be validated before proceeding. This could be used, for example, to ensure that the returned IP addresses are not part of the private address space (a DNS based blocking technique that is frequently used). What’s more, the urlgetter also allows us to use different DNS engines (e.g. we can use DNS over HTTPS).
In short, this basic building block allows us to decompose the steps of fetching a web page, observe and evaluate intermediate results, and decide how to move forward.
To understand how urlgetter could be very useful, let us consider the classic steps involved in fetching a web page over HTTPS, which are depicted in the following diagram:
miniooni research client, you can perform these steps as
./miniooni -i dnslookup://example.com urlgetter # TODO: extract and validate the returned IP addresses ./miniooni -ODNSCache="example.com 18.104.22.168" \ -i tlshandshake://example.com:443 urlgetter # TODO: store and inspect the returned certificate and/or # act upon errors ./miniooni -ODNSCache="example.com 22.214.171.124" \ -i https://example.com urlgetter # TODO: inspect the returned web page and compare w/ a # control measurement
Where the first step performs the DNS lookup, the second step does a TLS handshake using the IP address returned in the previous step, the third step fetches the web page.
Of course, we do not expect OONI Probe users to run these commands manually. Also, you may have noticed that the above script contains a bunch of TODOs. The aim of improving Web Connectivity’s methodology is indeed to fill in the gaps with smart logic aimed at detecting the several possible causes of blocking. Crucially, as mentioned above, we already determined that we wanted to include the SNI blocking methodology with every Web Connectivity run. This methodology could be be easily implemented using urlgetter by doing something like:
./miniooni -OTLSServerName="example.com" \ -i tlshandshake://helper.ooni.io:443 urlgetter
In this vein, we could also measure the consistency of the answer returned by the system DNS by forcing another DNS lookup using DNS over HTTPS, etc.
All in all, the availability of the urlgetter building block through miniooni allowed us to write Aladdin, a simple bash script that could implement all the steps we believe could be added to a future enhancement of Web Connectivity. This bash script is, of course, experimental, and we are planning on running further experiments like the ones described in this report, to continue refining our methodology. Once we are satisfied, we will rewrite the logic of this script in Go and integrate it into OONI’s measurement engine as the new Web Connectivity.
You can help us move this process forward by running Aladdin in interesting networks where you know there is blocking, and by discussing the results with us. In this regard, the best course of action is probably to join our Slack channels and coordinate with us.
Description of the Experiment
The version of aladdin that was used to measure www.womenonweb.org contained several checks, the most relevant of which are explained below. For further information, we encourage you to read the Aladdin script and, if in doubt, to ask us questions on our Slack channels.
One of the first steps performed by Aladdin was indeed to check for
SNI based blocking.
Since our SNI based blocking methodology connects to a test helper that
is unrelated to the domain we are testing, we expected this check to not
report any censorship for
www.womenonweb.org in the
specific ISP that exhibits the behaviour documented by Ververis et
We will refer to this as the
Because we knew there was TLS interference when connecting to the
www.womenonweb.org as the
TLS SNI, we also added a subsequent step where we did just
Incidentally, this step is what the Web Connectivity experiment that is
currently used by OONI Probe would already do. We expected this step to
fail, like it does for OONI Probe. We also wanted to assess whether it
collected useful extra evidence that is currently not collected by the
current version used by OONI Probe. We will refer to this as the
Lastly, we also wanted to fetch the block page returned by accessing
the website using HTTPS and by ignoring the invalid certificate
While we could conclude that there is something fishy going on by just
observing that the TLS handshake fails, the report from Ververis et
shows how the returned block page may potentially allow further
fingerprinting. We will refer to this as the
https_block page_fetch step.
Results Analysis & Discussion
The result of the
sni_blocking step confirms indeed that no
blocking occurs when we use the
www.womenonweb.org SNI with
an unrelated server used as a test helper.
This is the
miniooni command that the measurement script invoked:
./miniooni -v -A session=58908c79-865c-4c57-a6f1-ed59a5608e95 \ -A step=sni_blocking -O TLSServerName=www.womenonweb.org \ -i tlshandshake://126.96.36.199:443 urlgetter
example.org’s IP address,
-O TLSServerName forces the SNI, and all the other options are functional
to make the submitted measurement more actionable. (We added a
session annotation to track a batch of related measurements and a
step annotation so that we know very easily the intent of each
Below is a relevant excerpt from the measurement result (which is contained in the second line of the published report.jsonl file):
Under normal conditions, we expect this experiment to fail by telling
us that the
example.org server does not return a valid certificate
www.womenonweb.org website. This is exactly what the
ssl_invalid_hostname means in this context.
This finding is further confirmed by decoding from base64 and dumping the certificate included in the measurement:
% echo $certificate | base64 -d | openssl x509 -inform der -noout -text Certificate: Data: Version: 3 (0x2) Serial Number: 0f:d0:78:dd:48:f1:a2:bd:4d:0f:2b:a9:6b:60:38:fe Signature Algorithm: sha256WithRSAEncryption Issuer: C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA Validity Not Before: Nov 28 00:00:00 2018 GMT Not After : Dec 2 12:00:00 2020 GMT Subject: C=US, ST=California, [...], CN=www.example.org Subject Public Key Info: [...]
This clearly seems to be the certificate of
www.example.org. We can
thus conclude from this that there is no DPI rule in the network that
blindly matches the
www.womenonweb.org SNI. (We went on to manually
verify that the fingerprint of this certificate is indeed correct.)
Let us now focus on the
system_resolver_validation step. The
related command was:
./miniooni -v -A session=58908c79-865c-4c57-a6f1-ed59a5608e95 \ -A step=system_resolver_validation \ -O DNSCache="www.womenonweb.org 188.8.131.52" \ -i https://www.womenonweb.org/ urlgetter
Here we are configuring the DNS cache in such a way that
184.108.40.206 is returned as the IP address of
www.womenonweb.org, without making any network request. (This is
done because we have previously resolved
the system resolver and we want to reuse this result in this step,
without making an additional DNS request.)
We are marking this step as
220.127.116.11 was returned from the system resolver. Thus, if this
step succeeds, we can conclude that the IP address is either the address
of the server or the address of a transparent proxy that is letting us
reach the server without noticeably tampering with the TLS handshake.
As mentioned previously, this step is roughly equivalent to what OONI currently implements with its Web Connectivity test. It follows the system resolver results, it validates certificates, and it attempts to fetch the page hosted at the target URL. We expected this measurement to fail by saying that it encountered an invalid certificate, like Ververis et al. found.
This is the relevant excerpt from the related measurement (the seventh in the report.jsonl file):
Note how (1) we actually connect to the specified IP address and (2) the
TLS handshake fails after a bunch of messages are exchanged. This is
consistent with the
ssl_unknown_authority error, meaning that the
TLS handshake is completed, but the client doesn’t trust the server
certificate, because the signing authority is not recognized by the
client itself. (QED.)
Now, let us inspect the information that is automatically included in
this measurement and that was collected manually using
Ververis et al. For example, the rejected authority certificate looks
% echo $certificate | base64 -d | openssl x509 -inform der -text -noout Certificate: Data: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: C=ES, ST=Madrid, L=Madrid, O=Allot, OU=Allot, CN=allot.com/emailAddressemail@example.com Validity Not Before: Dec 16 13:07:49 2016 GMT Not After : Dec 16 13:07:49 2026 GMT Subject: C=ES, ST=Madrid, L=Madrid, O=Allot, OU=Allot, CN=allot.com/emailAddressfirstname.lastname@example.org [...]
The certificate seems to be signed by Allot, “a leading provider of innovative network intelligence and security solutions [...] powered by inline DPI technology [that] generates insightful intelligence that empowers our customers to optimize, innovate, and capitalize on every service opportunity”. Since Allot produces appliances to perform network intelligence, it seems plausible to speculate that Vodafone Spain may indeed be using such appliances to implement blocking. This corroborates the findings of Ververis et al.
Lastly, for the
https_block page_fetch step, we told
to accept any certificate returned by the server without validating it
(by using the
NoTLSVerify=true flag). The purpose of this experiment
is to fetch (a) the full certificate chain and (b) the returned block
page. (A future version of OONI Probe
Engine may be able
to perform this step and the previous step together by perhaps
delaying the verification of the certificate
./miniooni -v -A session=58908c79-865c-4c57-a6f1-ed59a5608e95 \ -A step=https_block page_fetch \ -O DNSCache="www.womenonweb.org 18.104.22.168" \ -O NoTLSVerify=true \ -i https://www.womenonweb.org/ urlgetter
Below is an excerpt from the measurement result (ninth line in the report.jsonl file):
Where (1) we clearly see from the network events that we arrive without errors to the end of the HTTP transaction (i.e. we fetch the web page); (2) we see that the server returns a small, possibly-binary block page; (3) we have the full certificate chain returned by the server.
If we decode the body from base64, we get this block page, which is consistent with the same block page mentioned in the report by Ververis et al:
<html><body><p>Por causas ajenas a Vodafone, esta web no está disponible</body></html>
Our (obvious) claim that the above is a block page, rather than the real home page of www.womenonweb.org, is further substantiated by the eight measurement, where we successfully fetch the real home page using Psiphon to evade censorship. (We do not show this measurement here for brevity; a real Web Connectivity measurement would have probably determined this by talking to our backend infrastructure instead.)
Let us instead focus on the certificate chain collected during this experiment. The following snippet shows part of the content of the first certificate in the chain:
% echo $certificate | base64 -d | openssl x509 -inform der -text -noout Certificate: Data: Version: 3 (0x2) Serial Number: 7b:25:9a:5e:00:00:00:00:d9:00:00:00:00:00:00:00 Signature Algorithm: sha256WithRSAEncryption Issuer: C=ES, ST=Madrid, L=Madrid, O=Allot, OU=Allot, CN=allot.com/emailAddressemail@example.com Validity Not Before: Apr 18 22:01:27 2019 GMT # 1 Not After : Apr 17 22:01:27 2021 GMT Subject: CN=www.womenonweb.org [...]
This certificate was created around a year ago (see annotation 1). Interestingly, it does not match the similar certificate published by Ververis et al, which was created on 27th January 2019. Nonetheless, this certificate and the one published by Ververis et al. clearly appear to be similar.
We confirm the findings by Ververis et al. They could have obtained the same data using an enhanced version of OONI Probe implementing the logic described in this report. The improvements we implemented enabled us to reproduce the findings by Ververis et al. by running a single command that automatically submits data to our collector.
We see that fetching the full certificate chain is much more informative than fetching the block page, but in other cases the block page could contain references to the vendor. Therefore, it seems that there is also some value in going all the way to fetching the block page.
The urlgetter experiment building block is valuable to us because it allows us to experiment with different logic for an improved implementation of OONI’s Web Connectivity test. (You can, by the way, help this process, by running the Aladdin bash script and discussing its results with us.)
It seems there are at least two ways to interfere with TLS. The case where filtering is applied for a specific SNI regardless of the server we are connecting to (as previously seen). And the case where, instead, blocking is only applied when using the right IP address (as seen here). Thus, our SNI blocking methodology, combined with ordinary Web Connectivity, allows us to conclude which blocking strategy for offending SNIs is actually implemented by ISPs.
In the coming weeks, we are going to perform more data analysis pertaining to results collected with Aladdin by volunteers that are already engaging with us. The result of this process will be an improved Web Connectivity test implemented in Go with enhanced capabilities, that will eventually be shipped as part of the OONI Probe apps so that people worldwide can easily run these experiments.