UPDATE (2023/03/11): After I received a comment from Yacoub (Thanks for leaving a comment in my “almost dead” blog!), I noticed Google has done some updated on their syntax for CT log… Hence I did a overnight update and I think my generator is working again (for Google’s Syntax V2). Unfortunately, Google did not update their script with the latest version, so I’m thinking of the options here… At least now, you can download the updated list from the table at the bottom of this post…
The download files are at the bottom
This is a tutorial about parsing a list of certificate transparency log servers (that are in OpenSSL requested formats)
If you use OpenSSL v1+, you might realize that openssl s_client
Some information about those flags:
-ct Request and parse SCTs (also enables OCSP stapling)
-noct Do not request or parse SCTs (default)
-ctlogfile infile CT log list CONF file
Basically, that’s a Certificate Transparency log that (mostly) CA embedded into your certificate, which is required by Chrome to not throw an error when you visit a site that has a certificate issued by this CA.
When you run the program with openssl s_client -ct
the first time (and every time you don’t have a CT log configuration file), you would (normally) expect an error like this
12580:error:02001002:system library:fopen:No such file or directory:crypto\bio\bss_file.c:74:fopen('C:\Program Files\Common Files\SSL/ct_log_list.cnf','rb')
12580:error:2006D080:BIO routines:BIO_new_file:no such file:crypto\bio\bss_file.c:81:
12580:error:0E078072:configuration file routines:def_load:no such file:crypto\conf\conf_def.c:150:
12580:error:3207B06D:CT routines:CTLOG_STORE_load_file:log conf invalid:crypto\ct\ct_log.c:209:
This is from a windows device.
Now, how to resolve this issue? (or, what is a ct_log_list.cnf file and how could i get one?)
If you dig on google.com, you would found those (somewhat unhelpful) lines: Link
# This file specifies the Certificate Transparency logs
# that are to be trusted.
# Google's list of logs can be found here:
# www.certificate-transparency.org/known-logs
# A Python program to convert the log list to OpenSSL's format can be
# found here:
# https://github.com/google/certificate-transparency/blob/master/python/utilities/log_list/print_log_list.py
# Use the "--openssl_output" flag.
Useless, hgh?
Some talent users.. (not me) might figure out how to do it…. or like me… stuck at one error forever.. and struggle to find the answer.
Let’s find a way to parse the log.. (If you don’t want to see the steps, scroll down to the bottom & I have one .cnf file parsed)
Prerequisites:
- have python,
wget , with command linecli (and could execute py files incommand line) - familiar with command line
- could install dependencies by yourself
- Linux Machine (or please use the parsed ct_log_list.cnf on the end of this tutorial)
Steps:
1. Go to the link.
You’ll need to see Google’s official repo before you download it… (make sure it’s still valid & exist)
Actually, go to this link https://github.com/google/certificate-transparency/tree/master/python/utilities/log_list since you would need to download all files from this folder.
2. Make a directory at a local machine (preferably Linux..) & download all ct tools from Google (Github repo)
Go to your machine, create one folder (name whatever you want)
Download all files in this folder
Commands to execute (Step 2)
cd ~/ #move to your home folder
mkdir ct_logs #make a directory called ct_logs, under your userhome
cd ~/ct_logs/
# Download all necessary python files from google ct tools
wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/print_log_list.py
wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/cpp_generator.py
wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/java_generator.py
wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/openssl_generator.py
3. Download ct_lists from Google (certificate transparency)
First, visit https://certificate.transparency.dev/useragents/, which has two agents (Google Chrome and Apple Safari). Both has a list of all logs their “agent” currently have & added
When you run print_all_log.py, it’s going to ask for several files, else it would throw an error.
List of files needed:
- log_list.json (with flag –log_list=log_list.json) (This is the file that contains all log servers included in chrome) or all_logs_list.json (with flag –log_list=all_logs_list.json) (all ct servers announced)
- log_list_pubkey.pem(with flag –signer_key=log_list_pubkey.pem) (Signature file, log_list.json ‘s signer public key)
- log_list.sig (with flag –signature=log_list.sig) (Signature file, prove that log_list is not modified)
Now, we could download all three files from http://www.certificate-transparency.org/known-logs (NOPE, they removed it).
Download log_list.json with this url: https://www.gstatic.com/ct/log_list/v2/log_list.json , log_list.sig from https://www.gstatic.com/ct/log_list/v2/log_list.sig , log_list_pubkey.pem from https://www.gstatic.com/ct/log_list/v2/log_list_pubkey.pem
Command to execute (Step 3)
cd ~/ct_logs/
# Download all necessary ct log & signature files from google
wget https://www.gstatic.com/ct/log_list/v2/log_list.json
wget https://www.gstatic.com/ct/log_list/v2/log_list.sig
wget https://www.gstatic.com/ct/log_list/v2/log_list_pubkey.pem
wget https://www.gstatic.com/ct/log_list/v2/all_logs_list.json
wget https://www.gstatic.com/ct/log_list/v2/log_list_schema.json
4. install all dependencies
Now you could run the python file
cd ~/ct_logs/
python print_log_list.py
You might see this error output:
Traceback (most recent call last):
File "print_log_list.py", line 11, in <module>
from absl import flags as gflags
ImportError: No module named absl
This means
pip install absl-py
Now run the script again, you would see another error
Traceback (most recent call last):
File "print_log_list.py", line 12, in <module>
import jsonschema
ImportError: No module named jsonschema
This means jsonschema package is missing, you would install it from pip
sudo pip install jsonschema
Run the script, yet another error
Traceback (most recent call last):
File "print_log_list.py", line 13, in <module>
import M2Crypto
ImportError: No module named M2Crypto
Install m2crypto from pip
sudo pip install m2crypto
After this, the script should be running, (but throw error like the following ones)
Traceback (most recent call last):
File "print_log_list.py", line 115, in <module>
sys.argv = FLAGS(sys.argv)
File "/home/pi/.local/lib/python2.7/site-packages/absl/flags/_flagvalues.py", line 633, in __call__
self._assert_all_validators()
File "/home/pi/.local/lib/python2.7/site-packages/absl/flags/_flagvalues.py", line 507, in _assert_all_validators
self._assert_validators(all_validators)
File "/home/pi/.local/lib/python2.7/site-packages/absl/flags/_flagvalues.py", line 528, in _assert_validators
raise _exceptions.IllegalFlagValueError('%s: %s' % (message, str(e)))
absl.flags._exceptions.IllegalFlagValueError: flag --log_list=None: Flag --log_list must be specified.
This means you are on the path, proceed to the next step
5. parse the log file
This would require a few flags with files, we’ve downloaded all in the previous steps.
Now, if you want to use the CT log server that
python print_log_list.py --log_list_schema log_list_schema.json --log_list log_list.json --signer_key log_list_pubkey.pem --signature log_list.sig --openssl_output ooo
Now, if you want to use all announced CT log server, run the below command, replace ‘
python print_log_list.py --log_list all_logs_list.json --signer_key log_list_pubkey.pem --signature log_list.sig --openssl_output ooo
If no errors are present, here you go..
You are done, open the file & place it to where your OpenSSL wants
No matter what file you downloaded, you’ll need to rename it to either ct_log_list.cnf or ct_log_list.conf, depend on what the OpenSSL wants.
The “all_list.conf” is a list of all known and announced CT Logs.
The “inc_list.conf” is a list of CT Logs that are currently compliant with Chrome’s CT policy (or have been and were disqualified), and are included in Chrome.
Provider | Description | Link |
Google/Chrome | All Known and Announced CT Logs (updated daily) | https://www.stevenz.download/ct-log/all_list.conf |
Google/Chrome | Chrome Approved & Included CT Logs (updated daily) | https://www.stevenz.download/ct-log/inc_list.conf |
Apple/Safari | Safari Approved & Included CT Logs (updated daily) | https://www.stevenz.download/ct-log/apple_inc_list.conf |
Have questions or feedback? Leave it in the comment area below
Hello Steven,
Thank you so much for the list as well as the steps that you provided. I got an error running python3 print_log_list.py
The error is:
FATAL Flags parsing error:
flag –log_list=None: Flag –log_list must have a value other than None.
flag –log_list_schema=None: Flag –log_list_schema must have a value other than None.
flags signature=None, skip_signature_check=None: Exactly one of (signature, skip_signature_check) must have a value other than None.
Pass –helpshort or –helpfull to see help on flags.
Do you have any idea on fixing this error.
Thank you.
Hi Yacoub,
Thanks again for reaching out!
As I promised in the email, I updated this page with a bit more updated instructions on how to generate the files, since Google decided to upgrade from schema v1 to v3. Link to Google Group: https://groups.google.com/a/chromium.org/g/ct-policy/c/zejEtWAJtEA/m/fX_AG-HUAQAJ
This means that the current OpenSSL print does not work (due to the new operator fields), but the good news is that Apple’s CT log list can also be generated with a minor update on the file named “openssl_generator.py” .
Just replace
def generate_openssl_conf(json_log_list, output_path):
'''Given a log list read from JSON, writes an OpenSSL log list to a file'''
with open(output_path, "w") as output:
logs = json_log_list["logs"]
log_confs = (_log_conf(log) for log in logs)
output.write(_enabled_logs_conf(logs) + "\n")
output.write("\n".join(log_confs))
with
def generate_openssl_conf(json_log_list, output_path):
'''Given a log list read from JSON, writes an OpenSSL log list to a file'''
with open(output_path, "w") as output:
logs = []
for op in json_log_list["operators"]:
logs = logs + op["logs"]
log_confs = (_log_conf(log) for log in logs)
output.write(_enabled_logs_conf(logs) + "\n")
output.write("\n".join(log_confs))
The primary addition is an empty dictionary of logs, and a new loop that goes through each of the operators field…
I bet there are much better or easier version of the same fix, but … I’m not a comp-sci major anymore (Technically I never am). So this is my layman’s fix.
I’ve also updated my list generator, so the list at the bottom of this page should be working correctly.
Now I’ll need to go subscribe on the CT-Policy google group so I won’t miss any schema updates…
Hope this works!
Sincerely,
Steven Zhu
Hello Steven,
Thank you so much. It is working now.
Best regards,
Yacoub.