BEATS 1— Aaron Aldrich - @CrayZeigh

THINGS WE’RE NOT TALKING ABOUT 2— Aaron Aldrich - @CrayZeigh

THINGS WE’RE NOT TALKING ABOUT ▸ root vegetables 2— Aaron Aldrich - @CrayZeigh

THINGS WE’RE NOT TALKING ABOUT ▸ root vegetables ▸ rhythm sections 2— Aaron Aldrich - @CrayZeigh

THINGS WE’RE NOT TALKING ABOUT ▸ root vegetables ▸ rhythm sections ▸ things by Dre 2— Aaron Aldrich - @CrayZeigh

Killer Tofu 3— Aaron Aldrich - @CrayZeigh

OK, WHAT ARE WE HERE FOR THEN? 4— Aaron Aldrich - @CrayZeigh

5— Aaron Aldrich - @CrayZeigh

KEEPING tempo WITH DATA 6— Aaron Aldrich - @CrayZeigh

HOW DOES IT ALL FIT TOGETHER? 7— Aaron Aldrich - @CrayZeigh

8— Aaron Aldrich - @CrayZeigh

HOW DO YOU MAKE A beat? 9— Aaron Aldrich - @CrayZeigh

10 — Aaron Aldrich - @CrayZeigh

START WITH A Base 11 — Aaron Aldrich - @CrayZeigh

libbeat FEATURES ▸ Processors ▸ Outputs ▸ Publisher Pipeline ▸ Configuration ▸ Logging ▸ Internal Metrics ▸ Audo Discovery ▸ CLI commands 12 — Aaron Aldrich - @CrayZeigh

Pick a flavor 13 — Aaron Aldrich - @CrayZeigh

▸ filebeat: logs, csv, text ▸ metricbeat: metrics ▸ packetbeat: transaction logs & flow metrics ▸ winlogbeat: Windows events ▸ auditbeat: system activity, changes ▸ heartbeat: network stats ▸ community beats: and a bunch of other stuff… 14 — Aaron Aldrich - @CrayZeigh

amazonbeatbeat, apachebeatbeat, apexbeatbeat, burrowbeat, cassandrabeat, cloudflarebeat, cloudfrontbeat, cloudtrailbeat, cloudwatchmetricbeat, cloudwatchlogsbeat, collectbeat, connbeat, consulbeat, dockbeat, elasticbeat, stcdbeat, execbeat, factbeat, flowbeat, gabeat, githubbeat, gpfsbeat, hsbeat, httpbeat, hwsensorsbeat, icingabeat, iobeat, jmxproxybeat, journalbeat, kafkabeat, krakenbeat, lmsensorsbeat, logstashbeat, mcqbeat, mongobeat, mqttbeat, mysqlbeat, nagioscheckbeat, nginxbeat, nginxupstreambeat, nvidagpubeat, openconfigbeat, packagebeat, phpfpmbeat, pingbeat, prombeat, prometheusbeat, protologbeat, redditbeat, redisbeat, retsbeat, rsbeat, saltbeat, springbeat, tracebeat, twitterbeat, udpbeat, udplogbeat, unifiedbeat, uwsgibeat, varnishlogbeat, varnishstatbeat, wmibeat HTTPS://WWW.ELASTIC.CO/GUIDE/EN/BEATS/LIBBEAT/CURRENT/COMMUNITY-BEATS.HTML 15 — Aaron Aldrich - @CrayZeigh

Personalize 16 — Aaron Aldrich - @CrayZeigh

Customization filebeat metricbeat winlogbeat packetbeat auditbeat heartbeat modules modules/metricsets event_logs protocol analyzers modules monitors 17 — Aaron Aldrich - @CrayZeigh

Cooking your own 18 — Aaron Aldrich - @CrayZeigh

19 — Aaron Aldrich - @CrayZeigh

DON’T FEAR THE GOPHER 20 — Aaron Aldrich - @CrayZeigh

libbeat CONFIGURATIONS FOR ALL BEATS 21 — Aaron Aldrich - @CrayZeigh

PUBLISHER PIPELINE 22 — Aaron Aldrich - @CrayZeigh

PUBLISHER PIPELINE 23 — Aaron Aldrich - @CrayZeigh

UNIVERSAL CONFIG name: ${hostname} tags: [“prod”, “web”] fields_under_root: false fields: {project: “my-project”, instance-id: “SOMEID”} 24 — Aaron Aldrich - @CrayZeigh

PREOCESSOR CONFIG processors: - ${processor_name} ${parameters}: when: ${condition} 25 — Aaron Aldrich - @CrayZeigh

PROCESSORS - add_cloud_metadata add_locale decode_json_fields drop_event drop_fields include_fields add_kubernetes_metadata add_docker_metadata 26 — Aaron Aldrich - @CrayZeigh

CONDITIONS - equals contains regexp range or and not 27 — Aaron Aldrich - @CrayZeigh

PROCESSOR CONFIG EXAMPLE processors: -drop_fields: fields: [provider_guid, process_id, thread_id, version, event_data.ErrorSourceTable] when: regexp: system.process.name: “foo.*” 28 — Aaron Aldrich - @CrayZeigh

OUTPUT CONFIG setup.template.settings: index.number_of_shards: 3 … output.elasticsearch: hosts: [“elastic:9200”] protocol: “https” username: “elastic” password: “changeme” index: “filebeat-%{[beat.version]}-%{+yyyy.MM.dd}” output.logstash: hosts: [“logstash:5044”] ssl.certificate_authorities: [“/etc/pki/root/ca.pem”] ssl.certificate: “/etc/pki/client/cert.pem” ssl.key: “/etc/pki/client/cert.key 29 — Aaron Aldrich - @CrayZeigh

Keystore metricbeat keystore create metricbeat keystore add output.elasticsearch.password output.elasticsearch: password: ${output.elasticsearch.password} 30 — Aaron Aldrich - @CrayZeigh

DASHBOARD CONFIGURATIONS $ .\filebeat setup -orsetup.dashboards.enabled: setup.kibana: host: “localhost:5601” protocol: “https” username: “elastic” password: “changeme” 31 — Aaron Aldrich - @CrayZeigh

Let’s cook 32 — Aaron Aldrich - @CrayZeigh

PUBLISHER PIPELINES 33 — Aaron Aldrich - @CrayZeigh

BEAT.CLIENT OPTIONS: ▸ Guaranteed = Do not drop events ▸ Sync = blocking publish ▸ Signal = (asynchronous) Signal callback 34 — Aaron Aldrich - @CrayZeigh

PROCESSORS - LIBBEAT type Processor interface { Run(event *beat.Event) (*beat.Event, error) String() string } 35 — Aaron Aldrich - @CrayZeigh

OUTPUTS - LIBBEAT type Client interface { Publish(publisher.Batch) error Close() error } type NetworkClient interface { Client Connect() error } 36 — Aaron Aldrich - @CrayZeigh

AUTODISCOVERY - LIBBEAT type ProviderBuilder func(bus.Bus, *common.Config) (Provider, error) type Provider interface { Start() Stop() } 37 — Aaron Aldrich - @CrayZeigh

GO modules 38 — Aaron Aldrich - @CrayZeigh

modules 39 — Aaron Aldrich - @CrayZeigh

modules ▸ Register in package ‘init’ ▸ Beats import module packages at build time 40 — Aaron Aldrich - @CrayZeigh

modules ▸ Register in package ‘init’ ▸ Beats import module packages at build time 41 — Aaron Aldrich - @CrayZeigh

modules ▸ Register in package ‘init’ ▸ Beats import module packages at build time ▸ Loader instatiates Module based on config 42 — Aaron Aldrich - @CrayZeigh

DICTIONARY STYLE modules output: elasticsearch: hosts: [‘localhost:9200’] ${module name}: ${module settings} 43 — Aaron Aldrich - @CrayZeigh

LIST STYLE modules metricbeat.modules: - module: ${module name} ${module settings} 44 — Aaron Aldrich - @CrayZeigh

modules CONFIG type moduleConfig struct { … } var defaultConfig = moduleConfig{ … } func init() { processors.RegisterPlugin(“name”, New) } 45 — Aaron Aldrich - @CrayZeigh

modules CONFIG func New(cfg *common.Config) (processors.Processor, error) { config := defaultConfig if err := cfg.Unpack(&config); err != nil { return nil, err } … } 46 — Aaron Aldrich - @CrayZeigh

MAIN-IMPORT TRICK package main import ( “os” “github.com/elastic/beats/libbeat/beat” “github.com/elastic/beats/metricbeat/beater” // load my plugins _ “my/module/package/name” ) func main() { if err := beat.Run(“mymetricbeat”, “”, beater.New); err != nil { os.Exit(1) } } 47 — Aaron Aldrich - @CrayZeigh

fields.yml 48 — Aaron Aldrich - @CrayZeigh

fields.yml INTRODUCTION For all events generated by beats we provide: ▸ Elasticsearch index template ▸ Kibana index pattern ▸ Documentation! ▸ Common definition and documentation in fields.yml 49 — Aaron Aldrich - @CrayZeigh

fields.yml SYNTAX field ::= name: <field name> type: <type> [format: <format>] description: <docstring> [fields: <fields list>] #’type’ must be group fields list ::= - <field> - <field> … type ::= group, long, keyword format ::= bytes, percent, … 50 — Aaron Aldrich - @CrayZeigh

fields.yml EXAMPLE - name: memory type: group description: > ‘memory’ contains local memory stats. fields: - name: total type: long format: bytes description: > Total memory 51 — Aaron Aldrich - @CrayZeigh

Explore the flavors 52 — Aaron Aldrich - @CrayZeigh

filebeat 53 — Aaron Aldrich - @CrayZeigh

FILEBEAT OVERVIEW 54 — Aaron Aldrich - @CrayZeigh

BASIC INPUT CONFIGURATION filebeat.prospectors: - type: log enabled: false paths: - /var/log/*.log #- c:\programdata\elasticsearch\logs* 55 — Aaron Aldrich - @CrayZeigh

FILEBEAT MODULES ▸ Pre-built set of configurations ▸ Simplify collecting, parsing, visualizing ▸ Modules per service ▸ Modules combine multiple filesets 56 — Aaron Aldrich - @CrayZeigh

MODULES CONFIG filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: true reload.period: 10s 57 — Aaron Aldrich - @CrayZeigh

INCLUDED MODULES filebeat/modules.d/ !”” apache2.yml.disabled !”” auditd.yml.disabled !”” icinga.yml.disabled !”” kafka.yml.disabled !”” logstash.yml.disabled !”” mysql.yml.disabled !”” nginx.yml.disabled !”” osquery.yml.disabled !”” postgresql.yml.disabled !”” redis.yml.disabled !”” system.yml.disabled #”” traefik.yml.disabled 58 — Aaron Aldrich - @CrayZeigh

ENABLING VIA COMMAND LINE $ ./filebeat modules enable nginx osquery auditd filebeat/modules.d/ !”” apache2.yml.disabled !”” auditd.yml !”” icinga.yml.disabled !”” kafka.yml.disabled !”” logstash.yml.disabled !”” mysql.yml.disabled !”” nginx.yml !”” osquery.yml !”” postgresql.yml.disabled !”” redis.yml.disabled !”” system.yml.disabled #”” traefik.yml.disabled 59 — Aaron Aldrich - @CrayZeigh

MODULE CONFIGURATIONS - module: nginx # Access logs access: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. #var.paths: # Error logs error: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. #var.paths: 60 — Aaron Aldrich - @CrayZeigh

AUTODISCOVER filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: kube-system config: - type: docker containers.ids: - “${data.docker.container.id}” exclude_lines: [“^\s+[-`(‘.|_]”] 61 — Aaron Aldrich - @CrayZeigh

drop asciiart lines

Let’s Cook 62 — Aaron Aldrich - @CrayZeigh

filebeat MODULES module/{module}/{fileset} !”” manifest.yml !”” config # $”” {fileset}.yml !”” ingest # $”” pipeline.json !”” _meta # $”” fields.yml $”” test 63 — Aaron Aldrich - @CrayZeigh

filebeat MODULES $ nginx/access/manifest.yml module_version: “1.0” var: - name: paths default: - /var/log/nginx/access.log* os.darwin: - /usr/local/var/log/nginx/access.log* os.windows: - c:/programdata/nginx/logs/access.log - name: pipeline default: ingest/pipeline.json ingest_pipeline: {{ .pipeline }} prospector: config/nginx-access.yml 64 — Aaron Aldrich - @CrayZeigh

filebeat IN SUMMARY Source Extension Points Use As Framework Publisher Guarantees On Back-Pressure log files inputs, filebeats modules yes depends, provided inputs: Send at least once wait/block 65 — Aaron Aldrich - @CrayZeigh

winlogbeat 66 — Aaron Aldrich - @CrayZeigh

winlogbeat OVERVIEW 67 — Aaron Aldrich - @CrayZeigh

CONFIG INPUTS PS C:\Users\vagrant> Get-EventLog * Max(K) Retain OverflowAction ——— ——— ——————-20,480 0 OverwriteAsNeeded 20,480 0 OverwriteAsNeeded 512 7 OverwriteOlder 20,480 0 OverwriteAsNeeded 20,480 0 OverwriteAsNeeded 20,480 0 OverwriteAsNeeded 15,360 0 OverwriteAsNeeded 68 — Entries ———75 0 0 0 1,609 1,184 464 Aaron Aldrich - @CrayZeigh Log —Application HardwareEvents Internet Explorer Key Management Service Security System Windows PowerShell

CONFIG INPUTS winlogbeat.event_logs: - name: Application ignore_older: 72h - name: Security - name: System 69 — Aaron Aldrich - @CrayZeigh

Can also specify channels directly PS C:> LogName LogName LogName LogName LogName LogName LogName LogName LogName LogName LogName LogName … Get-WinEvent -ListLog * | Format-List -Property LogName : Application : HardwareEvents : Internet Explorer : Key Management Service : Security : System : Windows PowerShell : ForwardedEvents : Microsoft-Management-UI/Admin : Microsoft-Rdms-UI/Admin : Microsoft-Rdms-UI/Operational : Microsoft-Windows-Windows Firewall With Advanced Security/Firewall 70 — Aaron Aldrich - @CrayZeigh

channel full name must be added to configuration winlogbeat.event_logs: - name: Microsoft-Windows-Windows Firewall With Advanced Security/Firewall 71 — Aaron Aldrich - @CrayZeigh

OTHER USEFUL WINDOWS CONFIG OPTIONS: winlogbeat.event_logs: - name: Security event_id: 4624, 4625, 4700-4800, -4735 # Can be filtered by event_id (white/blacklist) - name: Application provider: - Application Error - Application Hang - Windows Error Reporting # Can specify the log provider include_xml: true # Export raw XML available 72 — Aaron Aldrich - @CrayZeigh

winlogbeat IN SUMMARY Source Extension Points Use as framework Publisher Guarantees On back-pressure Windows Event Log API / Event logging API none not yet send at least once wait 73 — Aaron Aldrich - @CrayZeigh

metricbeat 74 — Aaron Aldrich - @CrayZeigh

metricbeat OVERVIEW 75 — Aaron Aldrich - @CrayZeigh

metricbeat MODULES ▸ Module ▸ group Metricsets ▸ matches Service Type ▸ provide common helper functions for Metricsets ▸ Metricset ▸ = Service Metrics Type ▸ Name matches Service API name 76 — Aaron Aldrich - @CrayZeigh

Configuration: DEFAULTS metricbeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false #reload.period: 10s setup.template.settings: index.number_of_shards: 1 index.codec: best_compression 77 — Aaron Aldrich - @CrayZeigh

Configuration: SYSTEM MODULE (IN METRICBEAT.YML) metricbeat.modules: - module: system metricsets: - cpu - filesystem - fsstat - load - memory - network - process - process_summary - uptime enabled: true period: 10s processes: [‘.*’] cpu.metrics: [“percentages”] # or ‘ticks’, ‘normalized_percentages’ core.metrics: [“percentages”] # or ‘ticks’ 78 — Aaron Aldrich - @CrayZeigh

modules.d is here again $ ./metricbeat modules enable system aerospike.yml.disabled ceph.yml.disabled docker.yml.disabled elasticsearch.yml.disabled golang.yml.disabled haproxy.yml.disabled jolokia.yml.disabled kibana.yml.disabled logstash.yml.disabled mongodb.yml.disabled nginx.yml.disabled postgresql.yml.disabled rabbitmq.yml.disabled system.yml vsphere.yml.disabled zookeeper.yml.disabled apache.yml.disabled couchbase.yml.disabled dropwizard.yml.disabled etcd.yml.disabled graphite.yml.disabled http.yml.disabled kafka.yml.disabled kubernetes.yml.disabled memcached.yml.disabled mysql.yml.disabled php_fpm.yml.disabled prometheus.yml.disabled redis.yml.disabled uwsgi.yml.disabled windows.yml.disabled 79 — Aaron Aldrich - @CrayZeigh

Configuration: DOCKER MODULE (IN MODULES.D) - module: docker metricsets: [“container”, “cpu”, “diskio”, “healthcheck”, “info”, “memory”, “network”] hosts: [“unix:///var/run/docker.sock”] period: 10s # To connect to Docker over TLS you must specify a client and CA certificate. #ssl: #certificate_authority: “/etc/pki/root/ca.pem” #certificate: “/etc/pki/client/cert.pem” #key: “/etc/pki/client/cert.key” 80 — Aaron Aldrich - @CrayZeigh

AUTODISCOVER etricbeat.autodiscover: providers: - type: docker templates: - condition: contains: docker.container.image: “redis” config: - module: redis metricsets: [“info”, “keyspace”] hosts: “${data.host}:6379” 81 — Aaron Aldrich - @CrayZeigh

Let’s Cook 82 — Aaron Aldrich - @CrayZeigh

metricbeat METRICSET API type EventFetcher interface { MetricSet Fetch() (common.MapStr, error) } type EventsFetcher interface { MetricSet Fetch() ([]common.MapStr, error) } 83 — Aaron Aldrich - @CrayZeigh

metricbeat IN SUMMARY Source Extension Points Use as Framework Publisher Guarantees On Back-Pressure Poll Services Modules and Metricsets Yes Drop after max_retries (default 3) wait 84 — Aaron Aldrich - @CrayZeigh

heartbeat 85 — Aaron Aldrich - @CrayZeigh

heartbeat OVERVIEW 86 — Aaron Aldrich - @CrayZeigh

heartbeat CONFIGURATION all done in heartbeat.yml heartbeat.monitors: - type: icmp schedule: ‘*/5 * * * * * *’ hosts: [“myhost”] … heartbeat.scheduler: limit: 10 87 — Aaron Aldrich - @CrayZeigh

OTHER MONITORS: TCP - type: tcp schedule: ‘@every 5s’ hosts: [“myhost”] ports: [80, 9200, 5044] ssl: certificate_authorities: [‘/etc/ca.crt’] supported_protocols: [“TLSv1.0”, “TLSv1.1”, “TLSv1.2”] 88 — Aaron Aldrich - @CrayZeigh

OTHER MONITORS: HTTP - type: http schedule: ‘@every 5s’ urls: [“https://myhost:80”] check.request: method: GET headers: ‘X-API-Key’: ‘12345-mykey-67890’ check.response: status: 200 body: ‘{“status”: “ok”}’ 89 — Aaron Aldrich - @CrayZeigh

Let’s Cook 90 — Aaron Aldrich - @CrayZeigh

heartbeat MONITOR API type Job interface { Name() string JobRunner } type JobRunner interface { Run() (beat.Event, []JobRunner, error) } 91 — Aaron Aldrich - @CrayZeigh

heartbeat IN SUMMARY Source Extension Points Use as Framework Publisher Guarantees On Back-Pressure Ping Services/Hosts Monitors Yes Drop after max_retries (default 3) wait 92 — Aaron Aldrich - @CrayZeigh

packetbeat 93 — Aaron Aldrich - @CrayZeigh

packetbeat OVERVIEW 94 — Aaron Aldrich - @CrayZeigh

packetbeat OVERVIEW protocol analyzer log transactions 95 — Aaron Aldrich - @CrayZeigh

packetbeat OVERVIEW flows collect connection metrics 96 — Aaron Aldrich - @CrayZeigh

packetbeat CONFIGURATION ▸ uses libpcap / WinPcap for network traffic capture ▸ need to configure listening device (specify or any) packetbeat.interface.device: en0 97 — Aaron Aldrich - @CrayZeigh

packetbeat PROTOCOL ANALYZERS packetbeat.protocols: - type: dns ports: [53] include_authorities: true include_additionals: true - type: http ports: [80, 8080, 8081, 5000, 8002] - type: memcache ports: [11211] - type: mysql ports: [3306] - type: tls ports: [443] - type: cassandra ports: [9042] 98 — Aaron Aldrich - @CrayZeigh

packetbeat FLOWS packetbeat.flows: timeout: 30s period: 10s # -1 reports killed only 99 — Aaron Aldrich - @CrayZeigh

Let’s Cook 100 — Aaron Aldrich - @CrayZeigh

packetbeat TCP INTERFACE API type TCPPlugin interface { Parse(pkt *Packet, address *common.TCPTuple, dir uint8, state ProtocolData) ProtocolData // common protocol settings getter GetPorts() []int ConnectionTimeout() time.Duration // connection FIN/packet loss ReceivedFin(…) ProtocolData GapInStream(…) (state ProtocolData, drop bool) } 101 — Aaron Aldrich - @CrayZeigh

packetbeat PROTOCOL ANALYZERS Code Generator: beats\packetbeats\scripts Contains Readme with sample tutorial Example: Cassandra by @medcl 102 — Aaron Aldrich - @CrayZeigh

packetbeat IN SUMMARY Source Extension Points Use as Framework Publisher Guarantees On Back-Pressure Network Packets Protocol Analyzers Yes Drop after max_retries (default 3) Transactions - Drop; Flows - wait 103 — Aaron Aldrich - @CrayZeigh

auditbeat 104 — Aaron Aldrich - @CrayZeigh

auditbeat OVERVIEW it’s basically metricbeat with custom modules 105 — Aaron Aldrich - @CrayZeigh

auditbeat OVERVIEW 106 — Aaron Aldrich - @CrayZeigh

auditbeat CONFIGURATION file_integrity auditbeat.modules: - module: file_integrity paths: - /bin - /usr/bin - /usr/local - /sbin - /usr/sbin - /usr/local/sbin - /etc 107 — Aaron Aldrich - @CrayZeigh

auditbeat CONFIGURATION auditd auditbeat.modules: - module: audtd audit_rules: | -w /etc/passwd -p wa -k identity -a always,exit -F arch=b32 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access 108 — Aaron Aldrich - @CrayZeigh

auditbeat IN SUMMARY Source Extension Points Use as Framework Publisher Guarantees On Back-Pressure Poll auditd, file system Modules Not yet Drop after max_retries (default 3) wait 109 — Aaron Aldrich - @CrayZeigh

Let me sum up 110 — Aaron Aldrich - @CrayZeigh

Let me sum up 111 — Aaron Aldrich - @CrayZeigh

Let me sum up Source Extension Points Use as Framework Publisher Guarantees Back-Pressure Handling FILEBEAT/WINLOGBEAT PACKETBEAT Log Files/Windows Events Network Packets Input & Modules Protocol Analyzers METRIC-/AUDIT-/HEARTBEAT Poll Services Modules/Metricsets/ monitors Yes/No Yes Yes Send-at-least-once (usually) Drop after N Retries Drop after N Retries Wait Transactions: Drop / Flows: Wait Wait 112 — Aaron Aldrich - @CrayZeigh

Demo! 113 — Aaron Aldrich - @CrayZeigh

WHOLE BEATS V. MODULES V. FORKED BEATS WHOLE BEATS MODULE BEAT Reason/Motivation Own Data Model/Logic Own Module/ Experiment Complexity/Flexibility High Low Maintenance moderate low Contribute Back Community list Community List/PR Packaging yes yes 114 — Aaron Aldrich - @CrayZeigh FORK BEAT Contribute Back Medium/Low depends PR yes

BATTERIES INCLUDED 115 — Aaron Aldrich - @CrayZeigh

Please attribute Elastic with link to Elastic.co Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders.