Security Congress Abstract
There is a firehose of information available to our security teams: logs, threat feeds, IOCs, data enrichment and more. One way to quickly search for that needle in the haystack is to create a "data lake." Using freely available tools, this presentation shows how we built a service that provides instant access to more than 30 billion detailed and enriched events per month. We found reliable methods utilizing Elasticsearch, Kafka and Grafana (along with other open source tools) to build a scalable and resilient solution. Over the course of two years of testing and research, we discovered what worked and what didn’t, and will share those experiences so you too can roll your own security data lake.