Crude Nginx Access Logs Parser

For the purposes of having fun with Go and not being bored, this is a very primitive Nginx access logs parser. It reads access logs and extracts the fields, writes it first to a .csv file and then to persistent .duckdb file. This simplifies further analysis. in Observable Framework.

Server Logs Extraction Process

cron job is setup on the VPS which copies nginx logs every night to the dedicated folder. On the client machine there is a bash script which wraps up the process as follows:

• copies files from the server to the client with rsync
• extracts and unpacks log files into a dedicated folder
• runs nginx-parser
• creates nginx.csv and nginx.duckdb
• copies nginx.duckdb to the Observable Framework project folder as well

This process is configured as systemd service and timer, which runs nightly.

Data Enrichment

• This repository is used as a source of IP addresses of various AI crawlers - Link
• This repository is used as a source of ASN information and IP ranges - Link