Geek asks and answers

Monitor Amazon S3 Logs with Splunk?

We have a large, expanded network of users that we track using icons. Total traffic is about 60 million impressions per month. We are currently considering moving from a rather slow, database-based logging solution (customizable with PHP-messy ...) to a simple database based on Amazon S3 and Splunk logs.

After using Splunk for some other analysis tasks, I really like it. But it is unclear how to configure a source such as S3 with the system. It seems that remote sources require Universal Forwarder to be installed, and this is not an option.

Any ideas on this?

+5
source share
3 answers

A very late answer, but I searched the same and found a Splunk application that does what you want, http://apps.splunk.com/app/1137/ . I haven't tried it though.

+1
source

I would suggest writing pre-processed j-son data to the documentdb database. For example, using azure queues or similar service bus messaging technologies that match your scenario in conjunction with azure documentdb. Therefore, I will support the database-based approach and modify it to be easy to scale the document-based database.

0
source

I use http://www.insight4storage.com/ from the AWS Marketplace to track the total usage of AWS S3 resources by prefix, bucket, or storage class over time; plus it shows me the previous version store with a prefix and for each bucket. It has the option to save S3 data as splunk logs, which may work in your use case, in addition to its user interface and web service API.

0
source

More articles:


All Articles