Just how can I parse Amazon S3 log documents?

What are the most effective alternatives for analyzing Amazon S3 (Simple Storage) log documents? I've activated logging and also currently I have log documents that resemble this:

858e709ba90996df37d6f5152650086acb6db14a67d9aaae7a0f3620fdefb88f files.example.com [08/Jul/2010:10:31:42 +0000] 68.114.21.105 65a011a29cdf8ec533ec3d1ccaae921c 13880FBC9839395C REST.GET.OBJECT example.com/blog/wp-content/uploads/2006/10/kitties_we_cant_stop_here_this_is_bat_country.jpg "GET /example.com/blog/wp-content/uploads/2006/10/kitties_we_cant_stop_here_this_is_bat_country.jpg HTTP/1.1" 200 - 32957 32957 12 10 "http://atlanta.craigslist.org/forums/?act=Q&ID=163218891" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.19) Gecko/2010031422 Firefox/3.0.19" -

What are the most effective alternatives for automating the log documents? I'm not making use of any kind of various other Amazon solutions apart from S3.

0
2019-05-07 01:31:18
Source Share
Answers: 1

Amazon advises making use of AWStats and also consists of instructions on how to set up an AWStats configuration file. You will certainly additionally require to set up some method to automate downloading and install the logs to where your AWStats installment is.

Directly I make use of cron to trigger a Python manuscript that makes use of boto to order the documents.

"""s3logDL.py

Downloads S3 logs and deletes old logs

Usage: python s3logDL.py"""

import sys
import os
from boto.s3 import Connection
import boto.s3

########################


########################

def main(argv):
    s3 = Connection('ACCESS_ID', 'PASSWORD')
    logsBucket = s3.get_bucket('logbucket')

    for b in logsBucket:
        #Download log
        print b.name
        key = logsBucket.get_key(b.name)
        fp = open("/PATH/TO/SAVE/POINT/" + b.name, "w")
        key.get_file(fp)

        #Delete log
        key.delete()

    exit(0)


########################

if __name__ == '__main__':
    main(sys.argv)
0
2019-05-08 22:16:46
Source