One of my script is scheduled to download everyday the proxy logs files from multiple proxies (Approx 1>2GB per file) of the previous day. The second step is to parse each of them and get the top 200 domain names within a specific environment. Finally at the end of the month another script create a report on the monthly internet usage.I thought this was an interesting exercise even if some tools would probably do a better job ($$$). Also we shouldn't take those results too seriously since some protocols like Ajax or HTML5 talk a lot to the servers, keep refreshing pages even if you are not actively working on them.
In this post, I will talk about the last part of this process and how I combine all those files to get a real monthly top domains.

