OpenSSL: ct_log_list.cnf

UPDATE (2023/03/11): After I received a comment from Yacoub (Thanks for leaving a comment in my “almost dead” blog!), I noticed Google has done some updated on their syntax for CT log… Hence I did a overnight update and I think my generator is working again (for Google’s Syntax V2). Unfortunately, Google did not update their script with the latest version, so I’m thinking of the options here… At least now, you can download the updated list from the table at the bottom of this post…

The download files are at the bottom

This is a tutorial about parsing a list of certificate transparency log servers (that are in OpenSSL requested formats)

If you use OpenSSL v1+, you might realize that your openssl s_client now has an option to parse CT logs from a certificate.

Some information about those flags:

-ct Request and parse SCTs (also enables OCSP stapling)
-noct Do not request or parse SCTs (default)
-ctlogfile infile CT log list CONF file

Basically, that’s a Certificate Transparency log that (mostly) CA embedded into your certificate, which is required by Chrome to not throw an error when you visit a site that has a certificate issued by this CA.

When you run the program with openssl s_client -ct the first time (and every time you don’t have a CT log configuration file), you would (normally) expect an error like this

12580:error:02001002:system library:fopen:No such file or directory:crypto\bio\bss_file.c:74:fopen('C:\Program Files\Common Files\SSL/ct_log_list.cnf','rb')
12580:error:2006D080:BIO routines:BIO_new_file:no such file:crypto\bio\bss_file.c:81:
12580:error:0E078072:configuration file routines:def_load:no such file:crypto\conf\conf_def.c:150:
12580:error:3207B06D:CT routines:CTLOG_STORE_load_file:log conf invalid:crypto\ct\ct_log.c:209: 
This is from a windows device.

Now, how to resolve this issue? (or, what is a ct_log_list.cnf file and how could i get one?)

If you dig on google.com, you would found those (somewhat unhelpful) lines: Link

# This file specifies the Certificate Transparency logs
# that are to be trusted.

# Google's list of logs can be found here:
# www.certificate-transparency.org/known-logs
# A Python program to convert the log list to OpenSSL's format can be
# found here:
# https://github.com/google/certificate-transparency/blob/master/python/utilities/log_list/print_log_list.py
# Use the "--openssl_output" flag.

Useless, hgh?

Some talent users.. (not me) might figure out how to do it…. or like me… stuck at one error forever.. and struggle to find the answer.

Let’s find a way to parse the log.. (If you don’t want to see the steps, scroll down to the bottom & I have one .cnf file parsed)

Prerequisites:

  1. have python, wget , with command line cli (and could execute py files in command line)
  2. familiar with command line
  3. could install dependencies by yourself
  4. Linux Machine (or please use the parsed ct_log_list.cnf on the end of this tutorial)

Steps:

1. Go to the link.

You’ll need to see Google’s official repo before you download it… (make sure it’s still valid & exist)

Actually, go to this link https://github.com/google/certificate-transparency/tree/master/python/utilities/log_list since you would need to download all files from this folder.

2. Make a directory at a local machine (preferably Linux..) & download all ct tools from Google (Github repo)

Go to your machine, create one folder (name whatever you want)

Download all files in this folder

Commands to execute (Step 2)

cd ~/    #move to your home folder
mkdir ct_logs      #make a directory called ct_logs, under your userhome
cd ~/ct_logs/
# Download all necessary python files from google ct tools
wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/print_log_list.py
wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/cpp_generator.py
wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/java_generator.py
wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/openssl_generator.py

3. Download ct_lists from Google (certificate transparency)

First, visit https://certificate.transparency.dev/useragents/, which has two agents (Google Chrome and Apple Safari). Both has a list of all logs their “agent” currently have & added

When you run print_all_log.py, it’s going to ask for several files, else it would throw an error.

List of files needed:

  1. log_list.json (with flag –log_list=log_list.json) (This is the file that contains all log servers included in chrome) or all_logs_list.json (with flag –log_list=all_logs_list.json) (all ct servers announced)
  2. log_list_pubkey.pem(with flag –signer_key=log_list_pubkey.pem) (Signature file, log_list.json ‘s signer public key)
  3. log_list.sig (with flag –signature=log_list.sig) (Signature file, prove that log_list is not modified)

Now, we could download all three files from http://www.certificate-transparency.org/known-logs (NOPE, they removed it).

Download log_list.json with this url: https://www.gstatic.com/ct/log_list/v2/log_list.json , log_list.sig from https://www.gstatic.com/ct/log_list/v2/log_list.sig , log_list_pubkey.pem from https://www.gstatic.com/ct/log_list/v2/log_list_pubkey.pem

Command to execute (Step 3)

cd ~/ct_logs/
# Download all necessary ct log & signature files from google
wget https://www.gstatic.com/ct/log_list/v2/log_list.json
wget https://www.gstatic.com/ct/log_list/v2/log_list.sig
wget https://www.gstatic.com/ct/log_list/v2/log_list_pubkey.pem
wget https://www.gstatic.com/ct/log_list/v2/all_logs_list.json
wget https://www.gstatic.com/ct/log_list/v2/log_list_schema.json

4. install all dependencies

Now you could run the python file

cd ~/ct_logs/
python print_log_list.py

You might see this error output:

Traceback (most recent call last):
  File "print_log_list.py", line 11, in <module>
    from absl import flags as gflags
ImportError: No module named absl

This means absl package is missing, you would install it from pip, if you don’t have pip and you don’t wish to install pip, compile absl manually from source https://github.com/abseil/abseil-py

pip install absl-py

Now run the script again, you would see another error

Traceback (most recent call last):
  File "print_log_list.py", line 12, in <module>
    import jsonschema
ImportError: No module named jsonschema

This means jsonschema package is missing, you would install it from pip

sudo pip install jsonschema

Run the script, yet another error

Traceback (most recent call last):
  File "print_log_list.py", line 13, in <module>
    import M2Crypto
ImportError: No module named M2Crypto

Install m2crypto from pip

sudo pip install m2crypto

After this, the script should be running, (but throw error like the following ones)

Traceback (most recent call last):
  File "print_log_list.py", line 115, in <module>
    sys.argv = FLAGS(sys.argv)
  File "/home/pi/.local/lib/python2.7/site-packages/absl/flags/_flagvalues.py", line 633, in __call__
    self._assert_all_validators()
  File "/home/pi/.local/lib/python2.7/site-packages/absl/flags/_flagvalues.py", line 507, in _assert_all_validators
    self._assert_validators(all_validators)
  File "/home/pi/.local/lib/python2.7/site-packages/absl/flags/_flagvalues.py", line 528, in _assert_validators
    raise _exceptions.IllegalFlagValueError('%s: %s' % (message, str(e)))
absl.flags._exceptions.IllegalFlagValueError: flag --log_list=None: Flag --log_list must be specified.

This means you are on the path, proceed to the next step

5. parse the log file

This would require a few flags with files, we’ve downloaded all in the previous steps.

Now, if you want to use the CT log server that are in Chrome, run the below command, replace ‘ooo‘ (after –openssl_output) with your OpenSSL ct config file path, or where you want the ct_log_list.cnf be placed

python print_log_list.py --log_list_schema log_list_schema.json --log_list log_list.json --signer_key log_list_pubkey.pem --signature log_list.sig --openssl_output ooo

Now, if you want to use all announced CT log server, run the below command, replace ‘ooo‘ (after –openssl_output) with your OpenSSL ct config file path, or where you want the ct_log_list.cnf be placed

python print_log_list.py --log_list all_logs_list.json --signer_key log_list_pubkey.pem --signature log_list.sig --openssl_output ooo

If no errors are present, here you go..

You are done, open the file & place it to where your OpenSSL wants

No matter what file you downloaded, you’ll need to rename it to either ct_log_list.cnf or ct_log_list.conf, depend on what the OpenSSL wants.

The “all_list.conf” is a list of all known and announced CT Logs.

The “inc_list.conf” is a list of CT Logs that are currently compliant with Chrome’s CT policy (or have been and were disqualified), and are included in Chrome.

ProviderDescriptionLink
Google/Chrome All Known and Announced CT Logs (updated daily)https://www.stevenz.download/ct-log/all_list.conf
Google/ChromeChrome Approved & Included CT Logs (updated daily) https://www.stevenz.download/ct-log/inc_list.conf
Apple/SafariSafari Approved & Included CT Logs (updated daily) https://www.stevenz.download/ct-log/apple_inc_list.conf

Have questions or feedback? Leave it in the comment area below

3 thoughts on “OpenSSL: ct_log_list.cnf”

  1. Hello Steven,

    Thank you so much for the list as well as the steps that you provided. I got an error running python3 print_log_list.py

    The error is:
    FATAL Flags parsing error:
    flag –log_list=None: Flag –log_list must have a value other than None.
    flag –log_list_schema=None: Flag –log_list_schema must have a value other than None.
    flags signature=None, skip_signature_check=None: Exactly one of (signature, skip_signature_check) must have a value other than None.
    Pass –helpshort or –helpfull to see help on flags.

    Do you have any idea on fixing this error.

    Thank you.

    1. Hi Yacoub,

      Thanks again for reaching out!
      As I promised in the email, I updated this page with a bit more updated instructions on how to generate the files, since Google decided to upgrade from schema v1 to v3. Link to Google Group: https://groups.google.com/a/chromium.org/g/ct-policy/c/zejEtWAJtEA/m/fX_AG-HUAQAJ
      This means that the current OpenSSL print does not work (due to the new operator fields), but the good news is that Apple’s CT log list can also be generated with a minor update on the file named “openssl_generator.py” .
      Just replace

      def generate_openssl_conf(json_log_list, output_path):
      '''Given a log list read from JSON, writes an OpenSSL log list to a file'''
      with open(output_path, "w") as output:
      logs = json_log_list["logs"]
      log_confs = (_log_conf(log) for log in logs)

      output.write(_enabled_logs_conf(logs) + "\n")
      output.write("\n".join(log_confs))

      with

      def generate_openssl_conf(json_log_list, output_path):
      '''Given a log list read from JSON, writes an OpenSSL log list to a file'''
      with open(output_path, "w") as output:
      logs = []
      for op in json_log_list["operators"]:
      logs = logs + op["logs"]
      log_confs = (_log_conf(log) for log in logs)

      output.write(_enabled_logs_conf(logs) + "\n")
      output.write("\n".join(log_confs))

      The primary addition is an empty dictionary of logs, and a new loop that goes through each of the operators field…
      I bet there are much better or easier version of the same fix, but … I’m not a comp-sci major anymore (Technically I never am). So this is my layman’s fix.

      I’ve also updated my list generator, so the list at the bottom of this page should be working correctly.

      Now I’ll need to go subscribe on the CT-Policy google group so I won’t miss any schema updates…

      Hope this works!

      Sincerely,
      Steven Zhu

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Scroll to Top