Making the OONI Probe Android app more resilient
We recently made OONI Probe Android more robust against accidental or deliberate blocking of our backend services. Specifically, we implemented support for specifying a proxy that speaks with OONI’s backend services. We also improved the build process to influence the TLS Client Hello fingerprint, which helps with avoiding accidental blocking.
Adding support for a proxy
Since late 2020, community members have been reporting specific OONI Probe Android failures. The symptom is that all tests fail with the “all available probe services failed” error. This issue has been publicly documented, for example, in ooni/probe#1324. The following screenshot shows what a user would see under this specific error condition.
The error in the above screenshot means that OONI Probe attempted to contact all the public OONI API endpoints (which we call “probe services”), but to no avail.
We first discussed implementing a proxy during the January 2020 OONI community meeting. After this meeting, we opened ooni/probe#985 to describe the implementation of this work. My initial thought on this matter was that we could automatically start a Psiphon tunnel upon failure.
But after discussing this with a few advanced users (thank you!), I refined my view on this issue. There will always be an advanced user who knows the local context better than us. These users will already have well-configured circumvention technology in place (e.g., an instance of Orbot running on the same device using bridges and pluggable transports).
These discussions and the urgency to solve the “all available probe services failed” issue also convinced me that we wanted to provide users with options first. Automatically detecting whether the ISP is censoring the probe services would then be implemented later.
The result of these conversations was a plan to allow users to choose whether they wanted to:
Use Psiphon; or
Manually specify a SOCSK5 proxy; or
Not use a proxy at all.
You can read through ooni/probe#985 and ooni/probe#1324 for more low-level details on what we did to make this possible. Here, for brevity, I will just mention the two most significant moments. On April 3, 2021, we added support for embedding an encrypted Psiphon config file into OONI Probe builds. This improvement enabled us to start a Psiphon tunnel without depending on the probe services for fetching this configuration file. On April 19, 2021, we discovered what was likely the root cause of the blocking. As I will further explain in the next section, it was the TLS Client Hello fingerprint. We needed to change how we build OONI Probe’s support libraries for Android devices to remediate this problem.
With these low-level details out of the way, my colleague Lorenzo and I started working on the user interface for configuring a proxy on Android (ooni/probe-android#423). A few Android users also assisted us, providing feedback, testing, and code patches.
This work resulted in adding an OONI backend proxy section in the settings of the OONI Probe app:
By tapping on the “OONI backend proxy” row, we will direct you to another screen, which allows you to choose which proxy you want to use:
The default (None) is that no proxy is being used. If you choose
Psiphon, OONI Probe will create a Psiphon
tunnel using the encrypted configuration file. If you select Custom,
you can set the hostname and port of a SOCKS5 proxy. I have filled the
hostname and port with 127.0.0.1 and 9050 in the above
screenshot. These are the settings you can use if you have an
Orbot
instance running on your device (or tor inside Termux). Instead, if you
are running Tor Browser,
the IP address is 127.0.0.1 and the port is 9150.
Changing our Android TLS fingerprint
In mid-April 2021, we discovered the likely cause of blocking. V2Ray developers previously documented this issue in v2fly/v2ray-core#557.
The root cause is that the TLS Client Hello generated by Golang on Android devices changes depending on Golang’s understanding of the hardware capabilities. Golang’s TLS library uses a hardware implementation of AES when possible. Otherwise, it falls back to AES code written in pure Go. According to the official docs, such code is not constant over time. Probably, for this reason, when Golang thinks that there is no hardware support for AES, it reorders the preferred cipher list in the Client Hello. This reordering aims to significantly reduce the probability that the server will select AES ciphers by moving such ciphers towards the bottom of the list.
As documented by V2Ray developers in the aforementioned issue, some ISPs only allow specific TLS Client Hello fingerprinting. When AES algorithms sit at the top of the preferred ciphers list, TLS works. Instead, when they are near the bottom, TLS fails.
We noticed this issue when debugging OONI Probe Android on one such
network. Very interestingly, the symptom was that a debug build of our
app was working. At the same time, a release build with “all the
available probe services" failed. Our analysis of
the problem is documented more in detail in the
ooni/probe#1444
issue. Golang fails to read the hardware capabilities because
/proc/self/auxv is not readable on Android in release mode.
(At the moment of writing this blog post, it is unclear whether
/proc/self/auxv is not readable on all arm64 Android devices or
whether we were just lucky with our devices.)
To fix this issue for OONI Probe, we forked Golang. We modified the src/runtime
to call the C library’s getauxval function to get the correct
hardware information. This patch seems a bit hacky and does not necessarily feel right in
general. Golang’s src/runtime should not depend on the C library.
Still, our solution works for us because we need to link the C library
on Android anyway.
When reviewing the diff, my colleague Arturo suggested that a better way to fix this issue is patching the golang/mobile repository instead. Patching golang/mobile feels like a more proper solution for this problem, but we have not had time to explore it yet.
For now, our mk build script uses ooni/go when building for Android.
While this may not be the best solution, it nonetheless provides us a
way to ship this crucial fix to users within a reasonable time frame.
Future improvements
Regarding the proxy functionality, we are also planning on implementing proxy support for OONI Probe iOS (ooni/probe#1470), the OONI Probe Command Line Interface (ooni/probe#1488), and the OONI Probe desktop app (ooni/probe#1489). We chose to prioritize Android because Android users have been impacted by this issue the most, as far as we know. If you see the “all available probe services failed” error on other platforms (beyond Android), please get in touch! (So far, we have received a single report of a similar problem occurring on iOS. If you also see this issue, please let us know by commenting on the ooni/probe#1491 issue.)
We are also planning on enabling users to choose HTTP proxies. We are considering the possibility of enabling users to specify a username and a password for SOCKS5/HTTP proxies. We are also discussing the option of passing Psiphon a downstream SOCKS5 proxy with optional authentication. We documented these future changes in ooni/probe#1465. If you have specific proxy needs, please let us know (perhaps on our Slack channel), and we will plan for them!
Regarding the fix for Golang, we need to figure out whether we can fix the issue inside of the golang/mobile repository (ooni/probe#1486). We also need to explore what happens on older arm devices (ooni/probe#1487). If you have insight on how we can fix the golang/mobile issue or ways to test old Android arm-based devices, please ping us!