Cooking with Elastic Beats

A presentation at Private Event in June 2018 in by fen aldrich

Slide 1

Slide 1

BEATS 1— Aaron Aldrich - @CrayZeigh

Slide 2

Slide 2

THINGS WE’RE NOT TALKING ABOUT 2— Aaron Aldrich - @CrayZeigh

Slide 3

Slide 3

THINGS WE’RE NOT TALKING ABOUT ▸ root vegetables 2— Aaron Aldrich - @CrayZeigh

Slide 4

Slide 4

THINGS WE’RE NOT TALKING ABOUT ▸ root vegetables ▸ rhythm sections 2— Aaron Aldrich - @CrayZeigh

Slide 5

Slide 5

THINGS WE’RE NOT TALKING ABOUT ▸ root vegetables ▸ rhythm sections ▸ things by Dre 2— Aaron Aldrich - @CrayZeigh

Slide 6

Slide 6

Killer Tofu 3— Aaron Aldrich - @CrayZeigh

Slide 7

Slide 7

OK, WHAT ARE WE HERE FOR THEN? 4— Aaron Aldrich - @CrayZeigh

Slide 8

Slide 8

5— Aaron Aldrich - @CrayZeigh

Slide 9

Slide 9

KEEPING tempo WITH DATA 6— Aaron Aldrich - @CrayZeigh

Slide 10

Slide 10

HOW DOES IT ALL FIT TOGETHER? 7— Aaron Aldrich - @CrayZeigh

Slide 11

Slide 11

8— Aaron Aldrich - @CrayZeigh

Slide 12

Slide 12

HOW DO YOU MAKE A beat? 9— Aaron Aldrich - @CrayZeigh

Slide 13

Slide 13

10 — Aaron Aldrich - @CrayZeigh

Slide 14

Slide 14

START WITH A Base 11 — Aaron Aldrich - @CrayZeigh

Slide 15

Slide 15

libbeat FEATURES ▸ Processors ▸ Outputs ▸ Publisher Pipeline ▸ Configuration ▸ Logging ▸ Internal Metrics ▸ Audo Discovery ▸ CLI commands 12 — Aaron Aldrich - @CrayZeigh

Slide 16

Slide 16

Pick a flavor 13 — Aaron Aldrich - @CrayZeigh

Slide 17

Slide 17

▸ filebeat: logs, csv, text ▸ metricbeat: metrics ▸ packetbeat: transaction logs & flow metrics ▸ winlogbeat: Windows events ▸ auditbeat: system activity, changes ▸ heartbeat: network stats ▸ community beats: and a bunch of other stuff… 14 — Aaron Aldrich - @CrayZeigh

Slide 18

Slide 18

amazonbeatbeat, apachebeatbeat, apexbeatbeat, burrowbeat, cassandrabeat, cloudflarebeat, cloudfrontbeat, cloudtrailbeat, cloudwatchmetricbeat, cloudwatchlogsbeat, collectbeat, connbeat, consulbeat, dockbeat, elasticbeat, stcdbeat, execbeat, factbeat, flowbeat, gabeat, githubbeat, gpfsbeat, hsbeat, httpbeat, hwsensorsbeat, icingabeat, iobeat, jmxproxybeat, journalbeat, kafkabeat, krakenbeat, lmsensorsbeat, logstashbeat, mcqbeat, mongobeat, mqttbeat, mysqlbeat, nagioscheckbeat, nginxbeat, nginxupstreambeat, nvidagpubeat, openconfigbeat, packagebeat, phpfpmbeat, pingbeat, prombeat, prometheusbeat, protologbeat, redditbeat, redisbeat, retsbeat, rsbeat, saltbeat, springbeat, tracebeat, twitterbeat, udpbeat, udplogbeat, unifiedbeat, uwsgibeat, varnishlogbeat, varnishstatbeat, wmibeat HTTPS://WWW.ELASTIC.CO/GUIDE/EN/BEATS/LIBBEAT/CURRENT/COMMUNITY-BEATS.HTML 15 — Aaron Aldrich - @CrayZeigh

Slide 19

Slide 19

Personalize 16 — Aaron Aldrich - @CrayZeigh

Slide 20

Slide 20

Customization filebeat metricbeat winlogbeat packetbeat auditbeat heartbeat modules modules/metricsets event_logs protocol analyzers modules monitors 17 — Aaron Aldrich - @CrayZeigh

Slide 21

Slide 21

Cooking your own 18 — Aaron Aldrich - @CrayZeigh

Slide 22

Slide 22

19 — Aaron Aldrich - @CrayZeigh

Slide 23

Slide 23

DON’T FEAR THE GOPHER 20 — Aaron Aldrich - @CrayZeigh

Slide 24

Slide 24

libbeat CONFIGURATIONS FOR ALL BEATS 21 — Aaron Aldrich - @CrayZeigh

Slide 25

Slide 25

PUBLISHER PIPELINE 22 — Aaron Aldrich - @CrayZeigh

Slide 26

Slide 26

PUBLISHER PIPELINE 23 — Aaron Aldrich - @CrayZeigh

Slide 27

Slide 27

UNIVERSAL CONFIG name: ${hostname} tags: [“prod”, “web”] fields_under_root: false fields: {project: “my-project”, instance-id: “SOMEID”} 24 — Aaron Aldrich - @CrayZeigh

Slide 28

Slide 28

PREOCESSOR CONFIG processors: - ${processor_name} ${parameters}: when: ${condition} 25 — Aaron Aldrich - @CrayZeigh

Slide 29

Slide 29

PROCESSORS - add_cloud_metadata add_locale decode_json_fields drop_event drop_fields include_fields add_kubernetes_metadata add_docker_metadata 26 — Aaron Aldrich - @CrayZeigh

Slide 30

Slide 30

CONDITIONS - equals contains regexp range or and not 27 — Aaron Aldrich - @CrayZeigh

Slide 31

Slide 31

PROCESSOR CONFIG EXAMPLE processors: -drop_fields: fields: [provider_guid, process_id, thread_id, version, event_data.ErrorSourceTable] when: regexp: system.process.name: “foo.*” 28 — Aaron Aldrich - @CrayZeigh

Slide 32

Slide 32

OUTPUT CONFIG setup.template.settings: index.number_of_shards: 3 … output.elasticsearch: hosts: [“elastic:9200”] protocol: “https” username: “elastic” password: “changeme” index: “filebeat-%{[beat.version]}-%{+yyyy.MM.dd}” output.logstash: hosts: [“logstash:5044”] ssl.certificate_authorities: [“/etc/pki/root/ca.pem”] ssl.certificate: “/etc/pki/client/cert.pem” ssl.key: “/etc/pki/client/cert.key 29 — Aaron Aldrich - @CrayZeigh

Slide 33

Slide 33

Keystore metricbeat keystore create metricbeat keystore add output.elasticsearch.password output.elasticsearch: password: ${output.elasticsearch.password} 30 — Aaron Aldrich - @CrayZeigh

Slide 34

Slide 34

DASHBOARD CONFIGURATIONS $ .\filebeat setup -orsetup.dashboards.enabled: setup.kibana: host: “localhost:5601” protocol: “https” username: “elastic” password: “changeme” 31 — Aaron Aldrich - @CrayZeigh

Slide 35

Slide 35

Let’s cook 32 — Aaron Aldrich - @CrayZeigh

Slide 36

Slide 36

PUBLISHER PIPELINES 33 — Aaron Aldrich - @CrayZeigh

Slide 37

Slide 37

BEAT.CLIENT OPTIONS: ▸ Guaranteed = Do not drop events ▸ Sync = blocking publish ▸ Signal = (asynchronous) Signal callback 34 — Aaron Aldrich - @CrayZeigh

Slide 38

Slide 38

PROCESSORS - LIBBEAT type Processor interface { Run(event *beat.Event) (*beat.Event, error) String() string } 35 — Aaron Aldrich - @CrayZeigh

Slide 39

Slide 39

OUTPUTS - LIBBEAT type Client interface { Publish(publisher.Batch) error Close() error } type NetworkClient interface { Client Connect() error } 36 — Aaron Aldrich - @CrayZeigh

Slide 40

Slide 40

AUTODISCOVERY - LIBBEAT type ProviderBuilder func(bus.Bus, *common.Config) (Provider, error) type Provider interface { Start() Stop() } 37 — Aaron Aldrich - @CrayZeigh

Slide 41

Slide 41

GO modules 38 — Aaron Aldrich - @CrayZeigh

Slide 42

Slide 42

modules 39 — Aaron Aldrich - @CrayZeigh

Slide 43

Slide 43

modules ▸ Register in package ‘init’ ▸ Beats import module packages at build time 40 — Aaron Aldrich - @CrayZeigh

Slide 44

Slide 44

modules ▸ Register in package ‘init’ ▸ Beats import module packages at build time 41 — Aaron Aldrich - @CrayZeigh

Slide 45

Slide 45

modules ▸ Register in package ‘init’ ▸ Beats import module packages at build time ▸ Loader instatiates Module based on config 42 — Aaron Aldrich - @CrayZeigh

Slide 46

Slide 46

DICTIONARY STYLE modules output: elasticsearch: hosts: [‘localhost:9200’] ${module name}: ${module settings} 43 — Aaron Aldrich - @CrayZeigh

Slide 47

Slide 47

LIST STYLE modules metricbeat.modules: - module: ${module name} ${module settings} 44 — Aaron Aldrich - @CrayZeigh

Slide 48

Slide 48

modules CONFIG type moduleConfig struct { … } var defaultConfig = moduleConfig{ … } func init() { processors.RegisterPlugin(“name”, New) } 45 — Aaron Aldrich - @CrayZeigh

Slide 49

Slide 49

modules CONFIG func New(cfg *common.Config) (processors.Processor, error) { config := defaultConfig if err := cfg.Unpack(&config); err != nil { return nil, err } … } 46 — Aaron Aldrich - @CrayZeigh

Slide 50

Slide 50

MAIN-IMPORT TRICK package main import ( “os” “github.com/elastic/beats/libbeat/beat” “github.com/elastic/beats/metricbeat/beater” // load my plugins _ “my/module/package/name” ) func main() { if err := beat.Run(“mymetricbeat”, “”, beater.New); err != nil { os.Exit(1) } } 47 — Aaron Aldrich - @CrayZeigh

Slide 51

Slide 51

fields.yml 48 — Aaron Aldrich - @CrayZeigh

Slide 52

Slide 52

fields.yml INTRODUCTION For all events generated by beats we provide: ▸ Elasticsearch index template ▸ Kibana index pattern ▸ Documentation! ▸ Common definition and documentation in fields.yml 49 — Aaron Aldrich - @CrayZeigh

Slide 53

Slide 53

fields.yml SYNTAX field ::= name: <field name> type: <type> [format: <format>] description: <docstring> [fields: <fields list>] #’type’ must be group fields list ::= - <field> - <field> … type ::= group, long, keyword format ::= bytes, percent, … 50 — Aaron Aldrich - @CrayZeigh

Slide 54

Slide 54

fields.yml EXAMPLE - name: memory type: group description: > ‘memory’ contains local memory stats. fields: - name: total type: long format: bytes description: > Total memory 51 — Aaron Aldrich - @CrayZeigh

Slide 55

Slide 55

Explore the flavors 52 — Aaron Aldrich - @CrayZeigh

Slide 56

Slide 56

filebeat 53 — Aaron Aldrich - @CrayZeigh

Slide 57

Slide 57

FILEBEAT OVERVIEW 54 — Aaron Aldrich - @CrayZeigh

Slide 58

Slide 58

BASIC INPUT CONFIGURATION filebeat.prospectors: - type: log enabled: false paths: - /var/log/*.log #- c:\programdata\elasticsearch\logs* 55 — Aaron Aldrich - @CrayZeigh

Slide 59

Slide 59

FILEBEAT MODULES ▸ Pre-built set of configurations ▸ Simplify collecting, parsing, visualizing ▸ Modules per service ▸ Modules combine multiple filesets 56 — Aaron Aldrich - @CrayZeigh

Slide 60

Slide 60

MODULES CONFIG filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: true reload.period: 10s 57 — Aaron Aldrich - @CrayZeigh

Slide 61

Slide 61

INCLUDED MODULES filebeat/modules.d/ !”” apache2.yml.disabled !”” auditd.yml.disabled !”” icinga.yml.disabled !”” kafka.yml.disabled !”” logstash.yml.disabled !”” mysql.yml.disabled !”” nginx.yml.disabled !”” osquery.yml.disabled !”” postgresql.yml.disabled !”” redis.yml.disabled !”” system.yml.disabled #”” traefik.yml.disabled 58 — Aaron Aldrich - @CrayZeigh

Slide 62

Slide 62

ENABLING VIA COMMAND LINE $ ./filebeat modules enable nginx osquery auditd filebeat/modules.d/ !”” apache2.yml.disabled !”” auditd.yml !”” icinga.yml.disabled !”” kafka.yml.disabled !”” logstash.yml.disabled !”” mysql.yml.disabled !”” nginx.yml !”” osquery.yml !”” postgresql.yml.disabled !”” redis.yml.disabled !”” system.yml.disabled #”” traefik.yml.disabled 59 — Aaron Aldrich - @CrayZeigh

Slide 63

Slide 63

MODULE CONFIGURATIONS - module: nginx # Access logs access: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. #var.paths: # Error logs error: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. #var.paths: 60 — Aaron Aldrich - @CrayZeigh

Slide 64

Slide 64

AUTODISCOVER filebeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.namespace: kube-system config: - type: docker containers.ids: - “${data.docker.container.id}” exclude_lines: [“^\s+[-`(‘.|_]”] 61 — Aaron Aldrich - @CrayZeigh

drop asciiart lines

Slide 65

Slide 65

Let’s Cook 62 — Aaron Aldrich - @CrayZeigh

Slide 66

Slide 66

filebeat MODULES module/{module}/{fileset} !”” manifest.yml !”” config # $”” {fileset}.yml !”” ingest # $”” pipeline.json !”” _meta # $”” fields.yml $”” test 63 — Aaron Aldrich - @CrayZeigh

Slide 67

Slide 67

filebeat MODULES $ nginx/access/manifest.yml module_version: “1.0” var: - name: paths default: - /var/log/nginx/access.log* os.darwin: - /usr/local/var/log/nginx/access.log* os.windows: - c:/programdata/nginx/logs/access.log - name: pipeline default: ingest/pipeline.json ingest_pipeline: {{ .pipeline }} prospector: config/nginx-access.yml 64 — Aaron Aldrich - @CrayZeigh

Slide 68

Slide 68

filebeat IN SUMMARY Source Extension Points Use As Framework Publisher Guarantees On Back-Pressure log files inputs, filebeats modules yes depends, provided inputs: Send at least once wait/block 65 — Aaron Aldrich - @CrayZeigh

Slide 69

Slide 69

winlogbeat 66 — Aaron Aldrich - @CrayZeigh

Slide 70

Slide 70

winlogbeat OVERVIEW 67 — Aaron Aldrich - @CrayZeigh

Slide 71

Slide 71

CONFIG INPUTS PS C:\Users\vagrant> Get-EventLog * Max(K) Retain OverflowAction ——— ——— ——————-20,480 0 OverwriteAsNeeded 20,480 0 OverwriteAsNeeded 512 7 OverwriteOlder 20,480 0 OverwriteAsNeeded 20,480 0 OverwriteAsNeeded 20,480 0 OverwriteAsNeeded 15,360 0 OverwriteAsNeeded 68 — Entries ———75 0 0 0 1,609 1,184 464 Aaron Aldrich - @CrayZeigh Log —Application HardwareEvents Internet Explorer Key Management Service Security System Windows PowerShell

Slide 72

Slide 72

CONFIG INPUTS winlogbeat.event_logs: - name: Application ignore_older: 72h - name: Security - name: System 69 — Aaron Aldrich - @CrayZeigh

Slide 73

Slide 73

Can also specify channels directly PS C:> LogName LogName LogName LogName LogName LogName LogName LogName LogName LogName LogName LogName … Get-WinEvent -ListLog * | Format-List -Property LogName : Application : HardwareEvents : Internet Explorer : Key Management Service : Security : System : Windows PowerShell : ForwardedEvents : Microsoft-Management-UI/Admin : Microsoft-Rdms-UI/Admin : Microsoft-Rdms-UI/Operational : Microsoft-Windows-Windows Firewall With Advanced Security/Firewall 70 — Aaron Aldrich - @CrayZeigh

Slide 74

Slide 74

channel full name must be added to configuration winlogbeat.event_logs: - name: Microsoft-Windows-Windows Firewall With Advanced Security/Firewall 71 — Aaron Aldrich - @CrayZeigh

Slide 75

Slide 75

OTHER USEFUL WINDOWS CONFIG OPTIONS: winlogbeat.event_logs: - name: Security event_id: 4624, 4625, 4700-4800, -4735 # Can be filtered by event_id (white/blacklist) - name: Application provider: - Application Error - Application Hang - Windows Error Reporting # Can specify the log provider include_xml: true # Export raw XML available 72 — Aaron Aldrich - @CrayZeigh

Slide 76

Slide 76

winlogbeat IN SUMMARY Source Extension Points Use as framework Publisher Guarantees On back-pressure Windows Event Log API / Event logging API none not yet send at least once wait 73 — Aaron Aldrich - @CrayZeigh

Slide 77

Slide 77

metricbeat 74 — Aaron Aldrich - @CrayZeigh

Slide 78

Slide 78

metricbeat OVERVIEW 75 — Aaron Aldrich - @CrayZeigh

Slide 79

Slide 79

metricbeat MODULES ▸ Module ▸ group Metricsets ▸ matches Service Type ▸ provide common helper functions for Metricsets ▸ Metricset ▸ = Service Metrics Type ▸ Name matches Service API name 76 — Aaron Aldrich - @CrayZeigh

Slide 80

Slide 80

Configuration: DEFAULTS metricbeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false #reload.period: 10s setup.template.settings: index.number_of_shards: 1 index.codec: best_compression 77 — Aaron Aldrich - @CrayZeigh

Slide 81

Slide 81

Configuration: SYSTEM MODULE (IN METRICBEAT.YML) metricbeat.modules: - module: system metricsets: - cpu - filesystem - fsstat - load - memory - network - process - process_summary - uptime enabled: true period: 10s processes: [‘.*’] cpu.metrics: [“percentages”] # or ‘ticks’, ‘normalized_percentages’ core.metrics: [“percentages”] # or ‘ticks’ 78 — Aaron Aldrich - @CrayZeigh

Slide 82

Slide 82

modules.d is here again $ ./metricbeat modules enable system aerospike.yml.disabled ceph.yml.disabled docker.yml.disabled elasticsearch.yml.disabled golang.yml.disabled haproxy.yml.disabled jolokia.yml.disabled kibana.yml.disabled logstash.yml.disabled mongodb.yml.disabled nginx.yml.disabled postgresql.yml.disabled rabbitmq.yml.disabled system.yml vsphere.yml.disabled zookeeper.yml.disabled apache.yml.disabled couchbase.yml.disabled dropwizard.yml.disabled etcd.yml.disabled graphite.yml.disabled http.yml.disabled kafka.yml.disabled kubernetes.yml.disabled memcached.yml.disabled mysql.yml.disabled php_fpm.yml.disabled prometheus.yml.disabled redis.yml.disabled uwsgi.yml.disabled windows.yml.disabled 79 — Aaron Aldrich - @CrayZeigh

Slide 83

Slide 83

Configuration: DOCKER MODULE (IN MODULES.D) - module: docker metricsets: [“container”, “cpu”, “diskio”, “healthcheck”, “info”, “memory”, “network”] hosts: [“unix:///var/run/docker.sock”] period: 10s # To connect to Docker over TLS you must specify a client and CA certificate. #ssl: #certificate_authority: “/etc/pki/root/ca.pem” #certificate: “/etc/pki/client/cert.pem” #key: “/etc/pki/client/cert.key” 80 — Aaron Aldrich - @CrayZeigh

Slide 84

Slide 84

AUTODISCOVER etricbeat.autodiscover: providers: - type: docker templates: - condition: contains: docker.container.image: “redis” config: - module: redis metricsets: [“info”, “keyspace”] hosts: “${data.host}:6379” 81 — Aaron Aldrich - @CrayZeigh

Slide 85

Slide 85

Let’s Cook 82 — Aaron Aldrich - @CrayZeigh

Slide 86

Slide 86

metricbeat METRICSET API type EventFetcher interface { MetricSet Fetch() (common.MapStr, error) } type EventsFetcher interface { MetricSet Fetch() ([]common.MapStr, error) } 83 — Aaron Aldrich - @CrayZeigh

Slide 87

Slide 87

metricbeat IN SUMMARY Source Extension Points Use as Framework Publisher Guarantees On Back-Pressure Poll Services Modules and Metricsets Yes Drop after max_retries (default 3) wait 84 — Aaron Aldrich - @CrayZeigh

Slide 88

Slide 88

heartbeat 85 — Aaron Aldrich - @CrayZeigh

Slide 89

Slide 89

heartbeat OVERVIEW 86 — Aaron Aldrich - @CrayZeigh

Slide 90

Slide 90

heartbeat CONFIGURATION all done in heartbeat.yml heartbeat.monitors: - type: icmp schedule: ‘*/5 * * * * * *’ hosts: [“myhost”] … heartbeat.scheduler: limit: 10 87 — Aaron Aldrich - @CrayZeigh

Slide 91

Slide 91

OTHER MONITORS: TCP - type: tcp schedule: ‘@every 5s’ hosts: [“myhost”] ports: [80, 9200, 5044] ssl: certificate_authorities: [‘/etc/ca.crt’] supported_protocols: [“TLSv1.0”, “TLSv1.1”, “TLSv1.2”] 88 — Aaron Aldrich - @CrayZeigh

Slide 92

Slide 92

OTHER MONITORS: HTTP - type: http schedule: ‘@every 5s’ urls: [“https://myhost:80”] check.request: method: GET headers: ‘X-API-Key’: ‘12345-mykey-67890’ check.response: status: 200 body: ‘{“status”: “ok”}’ 89 — Aaron Aldrich - @CrayZeigh

Slide 93

Slide 93

Let’s Cook 90 — Aaron Aldrich - @CrayZeigh

Slide 94

Slide 94

heartbeat MONITOR API type Job interface { Name() string JobRunner } type JobRunner interface { Run() (beat.Event, []JobRunner, error) } 91 — Aaron Aldrich - @CrayZeigh

Slide 95

Slide 95

heartbeat IN SUMMARY Source Extension Points Use as Framework Publisher Guarantees On Back-Pressure Ping Services/Hosts Monitors Yes Drop after max_retries (default 3) wait 92 — Aaron Aldrich - @CrayZeigh

Slide 96

Slide 96

packetbeat 93 — Aaron Aldrich - @CrayZeigh

Slide 97

Slide 97

packetbeat OVERVIEW 94 — Aaron Aldrich - @CrayZeigh

Slide 98

Slide 98

packetbeat OVERVIEW protocol analyzer log transactions 95 — Aaron Aldrich - @CrayZeigh

Slide 99

Slide 99

packetbeat OVERVIEW flows collect connection metrics 96 — Aaron Aldrich - @CrayZeigh

Slide 100

Slide 100

packetbeat CONFIGURATION ▸ uses libpcap / WinPcap for network traffic capture ▸ need to configure listening device (specify or any) packetbeat.interface.device: en0 97 — Aaron Aldrich - @CrayZeigh

Slide 101

Slide 101

packetbeat PROTOCOL ANALYZERS packetbeat.protocols: - type: dns ports: [53] include_authorities: true include_additionals: true - type: http ports: [80, 8080, 8081, 5000, 8002] - type: memcache ports: [11211] - type: mysql ports: [3306] - type: tls ports: [443] - type: cassandra ports: [9042] 98 — Aaron Aldrich - @CrayZeigh

Slide 102

Slide 102

packetbeat FLOWS packetbeat.flows: timeout: 30s period: 10s # -1 reports killed only 99 — Aaron Aldrich - @CrayZeigh

Slide 103

Slide 103

Let’s Cook 100 — Aaron Aldrich - @CrayZeigh

Slide 104

Slide 104

packetbeat TCP INTERFACE API type TCPPlugin interface { Parse(pkt *Packet, address *common.TCPTuple, dir uint8, state ProtocolData) ProtocolData // common protocol settings getter GetPorts() []int ConnectionTimeout() time.Duration // connection FIN/packet loss ReceivedFin(…) ProtocolData GapInStream(…) (state ProtocolData, drop bool) } 101 — Aaron Aldrich - @CrayZeigh

Slide 105

Slide 105

packetbeat PROTOCOL ANALYZERS Code Generator: beats\packetbeats\scripts Contains Readme with sample tutorial Example: Cassandra by @medcl 102 — Aaron Aldrich - @CrayZeigh

Slide 106

Slide 106

packetbeat IN SUMMARY Source Extension Points Use as Framework Publisher Guarantees On Back-Pressure Network Packets Protocol Analyzers Yes Drop after max_retries (default 3) Transactions - Drop; Flows - wait 103 — Aaron Aldrich - @CrayZeigh

Slide 107

Slide 107

auditbeat 104 — Aaron Aldrich - @CrayZeigh

Slide 108

Slide 108

auditbeat OVERVIEW it’s basically metricbeat with custom modules 105 — Aaron Aldrich - @CrayZeigh

Slide 109

Slide 109

auditbeat OVERVIEW 106 — Aaron Aldrich - @CrayZeigh

Slide 110

Slide 110

auditbeat CONFIGURATION file_integrity auditbeat.modules: - module: file_integrity paths: - /bin - /usr/bin - /usr/local - /sbin - /usr/sbin - /usr/local/sbin - /etc 107 — Aaron Aldrich - @CrayZeigh

Slide 111

Slide 111

auditbeat CONFIGURATION auditd auditbeat.modules: - module: audtd audit_rules: | -w /etc/passwd -p wa -k identity -a always,exit -F arch=b32 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access 108 — Aaron Aldrich - @CrayZeigh

Slide 112

Slide 112

auditbeat IN SUMMARY Source Extension Points Use as Framework Publisher Guarantees On Back-Pressure Poll auditd, file system Modules Not yet Drop after max_retries (default 3) wait 109 — Aaron Aldrich - @CrayZeigh

Slide 113

Slide 113

Let me sum up 110 — Aaron Aldrich - @CrayZeigh

Slide 114

Slide 114

Let me sum up 111 — Aaron Aldrich - @CrayZeigh

Slide 115

Slide 115

Let me sum up Source Extension Points Use as Framework Publisher Guarantees Back-Pressure Handling FILEBEAT/WINLOGBEAT PACKETBEAT Log Files/Windows Events Network Packets Input & Modules Protocol Analyzers METRIC-/AUDIT-/HEARTBEAT Poll Services Modules/Metricsets/ monitors Yes/No Yes Yes Send-at-least-once (usually) Drop after N Retries Drop after N Retries Wait Transactions: Drop / Flows: Wait Wait 112 — Aaron Aldrich - @CrayZeigh

Slide 116

Slide 116

Demo! 113 — Aaron Aldrich - @CrayZeigh

Slide 117

Slide 117

WHOLE BEATS V. MODULES V. FORKED BEATS WHOLE BEATS MODULE BEAT Reason/Motivation Own Data Model/Logic Own Module/ Experiment Complexity/Flexibility High Low Maintenance moderate low Contribute Back Community list Community List/PR Packaging yes yes 114 — Aaron Aldrich - @CrayZeigh FORK BEAT Contribute Back Medium/Low depends PR yes

Slide 118

Slide 118

BATTERIES INCLUDED 115 — Aaron Aldrich - @CrayZeigh

Slide 119

Slide 119

Slide 120

Slide 120

Please attribute Elastic with link to Elastic.co Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders.