- 129comments
- 22comments
- 35comments
- 38comments
- 15comments
- 259comments
- 10comments
- 2comments
- 12comments
- 2comments
- 3comments
- 88comments
- 101comments
- 2comments
- 13comments
- 823comments
- 288comments
- 6comments
- 13comments
- 1comments
- 58comments
- 75comments
- 248comments
- 95comments
- 1comments
- 16comments
- 30comments
- 261comments
- 33comments
- 69comments
do you guys just throw cloudflare in front and forget about it ? Or do you have engineers who work like data scientists - eyeball the logs ?
I have heard suggestions of using a firewall - but I'm genuinely curious on how do security focus companies discover things like "oh, we got attacked by Bitcoin miners from North Korea". Are there sophisticated tools that do this for you.. or is there a generally accepted practice that has evolved for even regular engineers to do this ?
P.S. I'm personally more interested in an API-focused answer, but I guess the same thing applies for websites.
fail2ban, but I'm not sure that's what you are looking for. You'd have to customize it for an API.
I assume that I suck at security, everything I have ever built is badly compromised and I'm too dumb to know it. I'm so bad at security that evil criminals actually celebrate my birthday.
Since I'm this bad, I assume that any monitoring tool I could install on the boxes themselves, up to and including all my logs are compromised. I'll usually still install monitoring tools for compliance, basic diligence and because they make wicked graphs that impress my superiors.
Since I'm that bad, I'd rather not invest 40 hours a week combing through fake logs unless I'm validating that I've taken care of the basic security fundamentals for my threat level.
Rather, I like to make sure that my security fundamentals are taken care of so that I can stop the simple attacks. If my threat/compliance model dictates, from there I'll possibly step things up into a web application firewall (WAF).
But, through all of this, I know that there are people out there who are so much smarter than me that the best I can do is little more than an annoyance for them. So, again, I like to assume that I'm badly compromised. This implies that I'm very careful about what data I keep, and I take efforts to protect my data at rest.
2nd'ing fail2ban, it's simple and awesome.
Beyond that, tripwire or it's modern equivalent. Traffic monitoring, "deep packet" inspection on the network side. Run all outbound through a proxy.
Not a lot of people seem to know what tripwire is nowadays.
There is this Linux thing called IMA which in some ways reminds me of tripwire (Integrity Measurement Architecture).
Nah, I’ve waned to use fail2ban multiple times and failed.
Unless something has recently changed it is some of the most poorly documented software on the planet.
I’ve never been able to configure the thing or figure out what it’s doing or... really anything besides find the homepage.
It’s also just not worth the trouble. Globally open SSH port? That’s like 1992 level Linux box in your parent’s basement stuff.
While it monitors SSH by default out of the box, people here are talking about creating custom filters which will monitor their Apache or Nginx logs. (Although, that said, we also have it monitor our non-standard SSH port with v. aggressive blocking rules, even though it's also behind a firewall. Defence in depth, and all that.)
The documentation isn't the greatest, but this page explains the concepts well enough and was easy to find using Google.
https://www.fail2ban.org/wiki/index.php/MANUAL_0_8
I understand the concepts but... how do you actually use it?
I like Fail2Ban, but as I remember it, my learning process looked a lot like this:
1.) Read through the documentation.
2.) Pour myself a scotch.
3.) Read through more documentation.
4.) Start drinking right out of the bottle.
5.) Try something.
6.) Get an error or experience a spectacular failure.
7.) Use StackOverflow.
8.) GOTO 5.
-------------
In good news, it worked - the light went on and now I'm confident that I could adapt Fail2Ban to suit any use. But, in bad news, honestly, I feel your pain and understand where/why you are getting stuck. Fail2Ban is a very complicated, yet extremely useful piece of software.
cd /etc/fail2ban and explore, you have example configs for all popular software and plenty of explanations in their comments
A filter matches lines in the nginx log using regular expressions. If the line matches, it uses another regexp to extract the IP address, and then calls out to scripts to block that IP address.
I'm not going to post the exact configuration files I use, but the GitHub repo for fail2ban contains examples.
https://github.com/fail2ban/fail2ban/blob/0.11/config/filter...
That's a filter that protects Apache against the 'shellshock' (https://en.wikipedia.org/wiki/Shellshock_(software_bug)) vulnerability, for example.
I use a home-grown solution called txrban:
http://www.kylheku.com/cgit/txrban/tree/
It consists of several processes written in the TXR language. One for monitoring Apache, one for sshd and one for Exim (mail server).
TXR's standard library has "tail streams": streams which automatically follow a rotating log file.
These txrban scripts simply open tail streams on their respective log files, use TXR pattern matching to scan for what they are a looking for, and then use functions in the tiny txrban utility library to do the banning. Banning is instigated by issuing iptables commands.
The banning isn't persisted, but when you restart the scripts, they retroactively replay everything by scanning the available logs from the beginning.
Banning events are reported via syslog. A "dry run" configuration variable allows testing for the rules without actually modifying your iptables.
I never packaged this program into a redistributable form. It contains some hard-coded log-file paths that work on my server, and no documentation.
I used to spend a lot of time combing my logs with custom scripts, but honestly I think the effort is better spent elsewhere. There're countless automated attacks a day on a big site but they really aren't anything you need to worry about.
most successful organizations ive seen don’t think about their API/app, but rather their stuff of value: usually data. If someone lifts your entire database, can you spot that on the way out the door? How quickly would you notice 100% of your fattest internet pipes spewing data at some poor souls website? You mention bitcoin; how long until you spot all your machines at 100% CPU?
Computers are better at the eyeballs-on-logs tasks. You teach them what is "normal" to be ignored and they may warn you about what is left, even the old logwatch is pretty capable for that.
Of course, there some basic security practices. Firewall, fail2ban or similar to stop/ban misbehaving IPs, monitoring systems metrics to detect abnormal activity/usage (not just for intrusion detection), and maybe feeding logs to i.e. ELK to notice new/unknown patterns.
Fail2ban looking for web pages that don't exist on your web site (like ../wpadmin) works pretty well at getting rid of script kiddies.
I suppose you could add a honeypot page that took form submissions and looked for any sequence attempting a CSRF, XSS, or SQL attack and ban the source there as well.
In short - security is such a big and multifaceted topic that it’s hard to know where to start, but I think that learning Kali Linux tools would cover a lot.
I started working at a cybersecurity company - our product is ridiculously stupid and I try and work as little as possible. In the time I’m not working I try to at least learn stuff. Since I’m at a cybersecurity place I sometimes try to learn about cybersecurity. It seems that even though for profit cs companies are making crazy dough, a lot of open source solutions out there do the same thing. There are so many attacks - they can be anywhere from the application level to the level of physical devices like routers having some kind of exploit. For any type of attack there is a good or multiple good open source tools that can be used to execute or defend (depending on what color hat you have), and a ton of them come standard installed on Kali Linux, but before you learn these you have to understand networks. As a python/Django dev I didn’t really have a deep knowledge and still don’t have a deep knowledge of this. Nor have I done more than dabble with any of these tools. But I know that there are a lot of videos on how to use these - including this 15 hour (!) one https://m.youtube.com/watch?t=1177s&v=vg9cNFPQFqM - if you were to go through that you’d probably be a security wizard
Thanks for the youtube link. Seems interesting. I am going to watch it in full.
I break into stuff professionally.
There is nothing I love more than the "just throw {Cloudflare,Akamai,Fastly,etc} in front of it" crowd. If you are going to deploy a WAF [1] it _needs_ to be at the edge of your datacenter or cloud environment. No matter what fancy feature your provider wants to sell you to fix this, there is a way to bypass it.
* Make sure you are logging centrally, and logging the right things.
* Make sure someone is looking at the logs, it doesn't have to be a full time person, but it needs to be someones job.
* Depending on how big you are, hire people to break in on some regular basis. It could be anything from hiring a cheap contractor to run a bunch of automated tools, to full on 10 person adversarial teams. They will find some stuff for you to fix, but more importantly you will know whether the first two points are working based on what you see.
1. https://www.owasp.org/index.php/Web_Application_Firewall
Could you please elaborate on what some of the right things generally are?
Okay this sounds incredibly interesting. Are there any open source groups that do this "for free" sorta? Not asking to get free working, asking so I could possibly jump in and learn.
we call it hacking forums/teams. main motivation is some idelogical/political belief tho.
imho try to hack yourself first
Speaking of "Hack Yourself First", this completely non-ideological (though somewhat commercial - it advertises the creator's Pluralsight course) website where you can learn and practice intro hacking skills:
https://hackyourselffirst.troyhunt.com/
There are websites that let you practice. hackthebox.eu is really popular. And there is a bit of a test to even get access to that.
Wow! Thanks for this! I "signed up" and it looks very interesting so far
If you're interested in learning the kind of particular skills used, there's a whole subculture in computer security having to do with Capture the Flags (CTFs), for a rather broad concept of "flag".
https://microcorruption.com/ is a quite a good one to get started with. (Because embedded systems are smaller and do less, they are easier to reason about, which makes it easier to see vulnerabilities.)
See also: https://www.reddit.com/r/securityCTF/
More broadly, the (with-permission) attackers are known as "the Red Team", and the concept is used against computer networks as well as in literal war-games by the US government.
Take the OSCP course, it is relatively cheap for the value of the content and it is one of the few certs in security that is respected.
From your profile it looks like you work for a big company. Ping some people in security and find out who runs your "Red Team" (if you have one). Make friends with them.
I disagree.
If your WAF is the edge of your datacentre, it will likely be ran by some sepperarte team with a generic ruleset, literally be MITMing all your connections, but more importantly it will be attackable via your company VPN or some other side channel like B2B connections generally, that you probably have no control over.
Your WAF, if you choose to have one, should sit directly in front of your app and have rules tailored to your app. Even those on VPNs, or in the datacentre , should be forced to traverse the apps WAF.
That said, the WAF logic is largely what you're going to do in your input validation layer, so unless you feel you need to double bag it try and to mitigate unknown framework or library issues, or are having another person write the rules, then you're just repeating work aimlessly.
When some dickhead manager buys an insecure http service they want on the web, you should probably quit. Failing that put a WAF in front of it.
It is up to you to define where your peremiter is, and put your protections there.
I generally recommend having the WAF live in front of of apps, but with a small gap that allows your scanners to still access the app. It is important to still be identifying vulnerabilities that your WAF might block, because they aren't perfect.
Just curious, how this is covered from legal standpoint? Do you ask clients to sign some consent document to release you from liability from damage from hacking by automated tools?
For work I am doing, most of the "automated tools" are things I wrote myself and I know what they do. It is up to the engineers working on the engagement to exercise care and avoid causing disruptions to the target or doing things they can't reverse easily.
I like to remind people that having someone on retainer that breaks your stuff with an automated tool is better than having some random guy on the internet break your site with the same tool. Either way it is eventually going to happen, but at least I will immediately notice and contact the right folks to help get it fixed.
Using Splunk to monitor traffic either via web logs or via more detailed packet capture solutions.
Just good web logs is enough to detect all kind of malicious anomalies.
Splunk machine learning functions can cluster data and detect anomalies in anything without rules or signatures:
https://imgur.com/a/IFhUZuH
You don't monitor web logs, you ship the logs to cold storage every [crontab spec units for time :)] and monitor your hosts and network for unusual activity.
If anything happens, THEN you look at the logs. The signal/noise ratio of the average web host logs makes combing through them just not worth it.
There are tons of products out there - have a look at OSSEC (https://www.ossec.net/), it's open source and free.
We also use Fail2Ban like a poor man's Web Application Firewall - for a Rails app you know you'll never have legit traffic to anything 'cgi' or 'php' or 'exe', so crafting a couple of regexps to match those and then ban them works well.
We use CloudFlare so we feed the IP bans into CloudFlare's firewall through their API as well as local IPTables. (We also use CloudFlare's WAF - we like redundancy in our paranoia.) If you had more demanding requirements you could look at mod_security for Apache or Nginx.
We also implement rate-limiting using Fail2Ban, which helps to trip the dumber automated scans.
Every time Fail2Ban blocks an IP address, it emails our admin team with the matching log entries and a whois on the IP address, which we then eyeball to make sure there's nothing weird going on.
TBH, easily 90% of the IP addresses we block got caught looking for 'wp-admin/admin.php', which is probably a good reason to never use WordPress.
I like this approach. Do you permaban the IPs? because a ton of ISPs rotate IPs across users
CloudFlare has a maximum number of firewall rules, based on the types of paid accounts you have. Every couple of months we purge the list when it gets close to being full.
99% of IP addresses we block are from outside our target market - we're a very local/regional company - so we don't really have to worry much about blocking legitimate customers. Very rarely our support team will get a request from someone that's been blocked and we'll unban the IP address, or set it to a JS/Captcha challenge at CloudFlare.
Just be aware when crafting those regexes that user-generated URLs (e.g. slug-based content paths, or even just query-string params if one is very careless) can thereby be used to build a denial of service attack against internal infrastructure, third-party integrations/webhooks, and even regular customers.
Reminds me that back in the day we used to be able to train email sanitizing appliances for false positives by sending them EICAR or GTUBE with spoofed headers.
We have definitely run into this, albeit only very very rarely. It's not viable for someone to build a usable DoS against us as an attack: our offices are on a whitelist and our set up prevents a viable attack against our internal infrastructure.
But yeah, like every tool, there's potential risks/down sides/things to watch for. There's no silver bullets!
Are these bots causing problems like consuming excess resources? If not it doesn't seem worth the effort to ban them.
These days I only block bots when they're e.g. filling out forms or posting junk comment that a human has to moderate.
Some kid running an automated vulnerability scan can be automatically detected and blocked this way, which can protect you if that scan would have succeeded otherwise. Of course, the proper way would be just to scan all that yourself and actually fix the problems, but that may be too difficult in some cases.
Scanning bots consume resources and traffic. I pay for both. They increase my bill so it makes sense to restrict them.
Having gone down this path before, I think time is better spent making your site more efficient so it doesn't matter. Developer time and site complexity are much more important commodities than bandwidth fees.
I usually leave it to the devops to deal with it. If it has to be a developer on my team to do it then you are right, I will spend their time elsewhere.
It seems you never had to deal with the amount of crap bots throw at every minor website out there.
If you don't want your logs flooded with vuln. scanning and referrer spam, use fail2ban or a waf.
I have, in fact, been doing this for years. Why should I care about having pristine logs?
Yeah, who cares about the extra disk space, the extra overhead, the waste of resources, filtering through noise to understand some issues with the app, right?
Isn't it lovely when every other referrer is trying to promote some online pharmacy or testing to see if you have a wp vulnerability?
Sure you can filter it, but I'd rather deal with garbage at the source.
A lesson I learned the hard way: if random bots are causing significant resourcing issues, then you need to address that directly. And if their impact isn't significant, why are you spending time on it? The disk space my log files consume is trivial.
While it's mostly bots trying to attack Wordpress, it's also blocked a few people spidering us aggressively (probably competitors) and other people trying attacks. Once it's scripted through a Chef cookbook, it's applied to all of our infrastructure with almost no effort, including any new boxes we launch. The biggest effort is running a script every few months to clear out the CloudFlare firewall, and I could probably stick that in a cron job, really.
Our technical team is very small and the company entirely dependent on our website and our reputation, so we take multiple approaches to securing the site and servers. Each little thing we add helps my boss sleep better, and that alone is probably worth the initial effort.
I definitely wouldn't do what some people are suggesting, manually reviewing logs/fail2ban matches and blocking IP addresses by hand. That's a waste of effort - and would probably be a full time job, too.
We also use fail2ban. It serves us good. When adding a new rule, we dont apply it but collect all the matched requests and ips. After 1 week, we review the log and if no customer traffic was captured, the rule is applied. We ban only for 30m. In my field, web, it takes 2m for a bot to disappear once its requests have been blocked. We dont use permanent bans as we havent had the need for them
Fail2ban is for all practical purposes resource depletion protection against bots that pose no threat but still generate work for the server. For this purpose you do sacrifice a bit of I/O performance, but hopefully won't get those spikes that brings things down.
It is not a replacement for the actually security of keeping things updated and having good passwords/client certificates.
I’ve learned that at the very least to ensure that everything is being properly backed up and that the restore works. Testing this regularly puts my mind at ease that even if my site, application, whatever gets destroyed, stolen, etc I can bring it back to life in a very short time. It’s very difficult to ensure that something is 100% secure (not even sure if possible since the major players like Twitter make mistakes). Of course, this doesn’t solve the issue of data being taken but that’s why it’s important to understand the data you actually store and if you truly need to store it or if some could be handled another way.
OSSEC is a nice FOSS tool for this
What is "an attack?" You need to define this better to get more focused answers.
Log monitoring is good for finding bad access patterns, but can't be trusted to find a problem as it happens.
Look for repeated errors. Hitting the same service over an over? Login attempt failures from the same few IPs? It's either a script, or the Bing crawler. Write application level code and network level configuration to mitigate the problem.
How about legitimate users doing social engineering attacks on each other? Sybil attacks in ratings or voting? Attempt to reuse mobile purchase tokens on different accounts? Some new SSHD credential leak zero day? Using your cat picture upload service for storing pictures of dogs? Each of these need different approaches, and a good logging system that can change with your needs is super helpful.
But you can't read the logs in real time, looking for bad patterns. That needs customized analytics.
OSSEC [1] is my go-tool for this type of tasks, it offers a language-agnostic interface to monitor any type of events. I used it along with my team to power the early version of the Sucuri Firewall, which later became the GoDaddy's security system. The project has matured a lot and is heavily tested using millions of HTTP requests every day. We also implemented a behavior-based algorithm to detect malicious requests, this part is not open-source but I can tell you that it was relatively easy to integrate it with the core interface.
Splunk [2] is also a good option, slightly more difficult to configure though. I have had the opportunity to meet some of the developers working in this company during a short visit to their Vancouver office and was genuinely surprised with the level of engineering that goes around this product. Definitely, worth checking it.
NewRelic [3] while the purpose has nothing to do with security, it also offers an easy-to-use interface to monitor events in the system. With some extra configuration you can make it work as a rudimentary firewall and benefit from all the great features that the product provides.
Disclaimer: I am an active contributor to the core features, but I am neutral on my recommendation.
[1] https://en.wikipedia.org/wiki/OSSEC
[2] https://en.wikipedia.org/wiki/Splunk
[3] https://en.wikipedia.org/wiki/New_Relic
1) fail2ban 2) log everything to an ELK stack (or your preferred alternative) and set up dashboards and reports to feed you malicious crafted URLs 3) Reject traffic in your web proxy based on a maintained list of hostile URLs. Ban the source of this traffic. Don't send back 500s, just slow down and eventually stop answering the requests. HAproxy lets you do this easily with a few simple rules and a stick table.
The answer isn't quite "don't bother" but it's definitely not "install these tools and get started".
Most of the things you can easily set up to watch for security events are looking for problems you simply shouldn't have.
For instance: you can set up fail2ban, sure. But what's it doing for you? If you have password SSH authentication enabled anywhere, you're already playing to lose, and logging and reactive blocking isn't really going to help you. Don't scan your logs for this problem; scan your configurations and make sure the brute-force attack simply can't work.
The same goes for most of the stuff shrink-wrap tools look for in web logs. OSSEC isn't bad, but the things you're going to light up on with OSSEC out of the box all mean something went so wrong that you got owned up.
Same with URL regexes. You can set up log detection for people hitting your admin interfaces. But then you have to ask: why is your admin interface available on routable IPs to begin with?
What you want is an effectively endless sink for developer logs (Splunk is fine but expensive, ELK is fine but complicated to manage), and you want to add structured instrumenting logs to your applications in all the security-sensitive places, like authorization checks (which should rarely fail), and especially a very solid, very long-term audit trail of admin-like actions, so you have some prayer of tracking insider misuse. You could instrument your SQL queries and catch any query failures --- which should also be rare. That kind of stuff. I don't think there's a good shrink-wrap package that does that.
But really: focus on removing attack surface, rather than continuously monitoring it.
You buried the lede on this one.
I think you meant “Splunk” :)
thx
Or, you know, install these tools and get started:
Don't do most of this.
Care to elaborate?
You almost certainly don't want to set up a WAF.
Don't use any of those scanners, all of which look almost exclusively for vulnerabilities you don't have. Do look into getting a Burp license for every developer on your team.
By all means, harden Nginx. Don't pay SpiderLabs to do it for you.
A WAF will stop a lot of the stupid junk that a kid with a "how 2 hax0r" book can make work a disturbing amount of the time.
If any of that stuff is actually working on your site, not having a WAF is the least of your problems. Don't waste time with this stuff.
I assume you are writing in an attempt to persuade.
Your message is being lost/ignored as a result of your delivery.
We want to understand your message, but you aren't giving enough background. I don't trust your message because there is no supporting background. If you're going to tell me to not do something, either:
A) provide a compelling alternative (not doing $it is not a compelling alternative to $it in nearly all cases) B) explain in detail why you shouldn't do $it. Provide more details if asked.
I want to believe what you are saying and be convinced by it, please help me. (Note: I'm not trying to be a dick, I've just known a lot of people who are perpetually ignored because of this communication style and take forever to learn that lesson because no one bothers to say anything)
Is this advice assuming that it's devs who want the WAF, rather than ops or management?
For devs clearly it's not what they should be focusing on. But saying something isn't worth doing because it shouldn't work is a terrible security principle.
A WAF will also add an attack surface, just like anti-virus software before them.
Company X, a content publisher, has a wordpress site with a bunch of plugins. They hear that the security record of such a setup is less than ideal. They buy a WAF solution to protect them.
Company Y, also a content publisher in the same space, realizes that they don't need dynamically generated content. They don't need tons of JS. They start publishing static content and what little dynamic functionality they have is well-compartmentalized, with interactions to the outside world carefully audited.
Company Y could also get a WAF, but why? It increases the attack surface, Company Y probably don't have the time/expertise to audit it, it is not clear what benefits it will have.
This is brilliant.
Thanks !
There are good burp alternatives too :)
There definitely are! Your stuff included! Burp is just the industry standard.
So is it "fix whatever Burp points out" ? Is that a good first step ? Especially for api endpoints (versus websites)
The problem with throwing tools at the thing is you have no idea what it's doing or how it works or IF it works, and it becomes forgotten about. But it's better than nothing in the short term while you figure out how to secure your application. Slap ModSecurity in front of your app while you go over OWASP and managing cloud infrastructure securely.
In addition to the points discussed..
If you follow the Geekflare link to modsecurity, and then follow the link there to a download page, there's this warning:
That's not something I would suggest throwing in production, given that warning has been there for years.
The post itself describes Shadow daemon as "probably defunct".
Ironbee's last commit was two years ago and I get an nxdomain looking for their website.
AWS WAF is both expensive and and extremely manual. You won't get anything out of it without a major labour investment.
Nginx's built in WAF is only in the pro edition, and outside the price of many people.
This whole discussion however depends on your customer. I have an agency I support where I have to provide a report on every single hack attempt. That means if my Rails application gets hit with said three year old Wordpress exploit, and the web server "404", I have to report on how we "blocked" it. And no, "not applicable" is not an answer.
So we throw money at commercial products. And sometimes I recommend doing so. But like security can involve just being aware of your risks, this use of a WAF is about being aware of the real problem you're solving, which is a paper work based one.
Is there a course (or set of courses), books or other learning material you recommend for web developers to get to a competent level in terms of building, deploying and maintaining secure applications (including infrastructure)?
Edit: updated to be more clear
Get more into DevOps/SRE/Ops/SysAdmin/WordOfTheWeek where building secure infrastructure is part of their job. Writing secure applications is one thing, deploying them on secure infrastructure is another.
Thanks for the tips!
Yeah, I should have been more explicit. When I say build secure applications, I imply the full stack including infrastructure. I updated my comment accordingly.
It is a good advice but the OP is asking for web log files not ssh and any other logs. You cannot remove the attack surface when the app/service is served via http
If you use a service such as Cloudflare, you can restrict the ip range, ensuring that all requests go through that service (which probably has more experience with malicious requests than the website owner).
You cannot remove the attack surface when the app/service is served via http
Remove? Probably not. There are many options for minimizing the attack surface with http.
I came here to add an important detail about WAFs that's missing in this thread. For context, I am the original author of ModSecurity (but haven't worked on it since at least 2009). I wrote ModSecurity for three use cases that still make a lot of sense sense; they are: 1) full HTTP transaction logging (meaning request and response bodies), 2) rule-based analysis of HTTP traffic (largely to trigger logging and otherwise highlight anomalies) and 3) virtual patching.
The first two use cases are all about traffic visibility, which has been, and remains a big problem. If we're talking about improving security, the third use case—virtual patching—is the exciting part. The story goes something like this: you discover a vulnerability in your code but you can't fix it immediately, for whatever reason (take your pick). In practice, it may be weeks until the problem is fixed. However, if you have ModSecurity installed, you can write a virtual patch for your problem, often to fix it 100% reliably.
However, it's very important to understand that virtual patching doesn't mean just slapping together some rule that appears to be doing the job. Blacklisting doesn't work. To properly virtual-patch, you have to fully understand the vulnerability and the entry point to the application. You have to carefully whitelist all parameters in that particular location. In other words, you have to almost replicate the work that should have been done by the developer.
Even in this context, it's not all rosy. Virtual patching requires a great deal of expertise to do properly. It's almost always better to spend your time fixing the actual problem. Sometimes, however, that's not possible to do in a reasonable time frame, and that's when a WAF (according to my definition), can be a life saver.
Thanks for this. I admire ModSecurity! But the subtext of my comment is the needs of typical HN people, most of whom manage applications they wrote, for small companies. Overwhelmingly, those people should spend any time they would spend on things like ModSecurity instead shoring up the security of their code.
TLDR: Ivan, thank you for everything you've provided to the IT community.
- - - -
There's few people I am more pleased to see commenting here on a topic like Sec more than yourself Ivan, your work's exceptional. And the tools you've provided like Hardenize and SSL Labs have saved me countless hours.
Your books have been an immense source of knowledge for me (and I'm relatively a newcomer by my colleagues standards).
Really pleased to see you here on HN. Though, I'm not entirely surprised. The best of the best seem to haunt this place.
I use some custom fail2ban rules on my web servers, for example too many hits on url X in a 10 minute window = firewalled for a day. Helps stop brute forcing on web apps e.g. Wordpress and can cut down massively on the traffic the server is having to deal with.
I Also have some mod_rewrite rules in to 403 some URL’s that bots currently hit, mostly because I don’t want to spend 200ms of PHP time for the bot to discover that it’s exploit didn’t work.
I accept that when you’re hosting websites for a load of other people you’re gonna need backups too, that’s the bottom line.
I don't even give them the courtesy of a 403. That's more information than they deserve. HAproxy severely throttles their requests, at no cost to my web servers, and eventually stops responding entirely.
Converting our WordPress sites to static sites greatly simplified and improved our infrastructure.
You have railed against network isolation as defense in other threads. Locally-scoped IPs, VPNs, and ACLs are all about network isolation. Care to shed some light on the inconsistency?
I did?
The parent posted what is pretty much the right answer, but to add:
* Your logs should be there to help you find out what happened in the event of a compromise, not to detect it in progress.
What OP should be monitoring is the activity that occurs on the system - logins, processes spawned (e.g. php or mysql suddenly spawning shells for no reason), changes to content, resource volume spikes and as the parent said, configuration changes.
Much of this can be done with a variety of tools but there isn't really a one-size fits all approach. I'd start with resource spikes as it's easiest and work up to an OSSEC configuration reflecting the actual environment.
Also by way of middle ground on the log front, Graylog2 is a happy medium between Splunk and ELK, but it's better to log fewer (but more useful) things than a load of noise.
In my experience, monitoring this takes about as much effort as adding a new apparmor / selinux / firejail / ... rule which would block it in the first place. I mean, with processes spawned we're almost at auditd level already.
I vehemently disagree with this assertion. I also disagree with tptacek's asseration that any admin interface should be inaccessble from the public Internet. That's just not how a lot of small businesses, non-profits, etc. operate. 2FA? Sure. Geo-fencing? Keeps out noise. Basic auth / some front-end to minimize app-specific attack surface? Perhaps.
Saying "you shouldn't look at logs to detect anomalous activity before the fact" is awful. So is "If you have and admin availability publicly, you're screwed".
My saying is "Security is a spectrum". A bank should not have their firewall admin interface publicly available. A community newsletter blog would probably be alright.
I don't know what the maturity cutoff you're using here is, but I'll draw my line at "deploying on AWS". If you're on AWS, you have really no business exposing an admin interface on a routable address. Deploy it on an internal address and have admins use a VPN to reach it.
There are organizations on AWS that have publicly-routable admin interfaces. They should all be working on plans to migrate away from that, and, in the meantime, they should be extra careful about authentication and monitoring.
If you're too small to have a non-routable admin interface, you are definitely too small to pay for security products.
Mixing discussions between people makes it harder to respond to your comments. It's better to have one comment aimed at me and one aimed at tptacek.
As for the content of your comment, it's hard to distinguish what's aimed at me and what's aimed at him, so I'll assume everything aside from your first sentence is aimed at him.
This applies if you are developing your own software.
If you do security for an enterprise with dozens or hundreds of apps that you don't develop yourself, you might not have the luxury of only trying to reduce attack surface, or instrumenting 3rd-party supplied code.
Yes, that's true.
Thanks for this. Just removed ssh login from a server.
The suggestion was to only allow logins with ssh keys - not disable ssh logins entirely.
In an ideal world, literally no SSH at all. That's doable in container cattle fleet networks.
In a realistically achievable world for virtually every startup, no routable SSH. No security rule allows 22/tcp from a routable address into a VPC. You want to SSH into a host, you VPN to your network first. This is how most organizations with security teams set up SSH.
If you can't get there, at least get to jump box SSH --- 22/tcp is allowed to exactly one isolated host from the outside world (some people filter source addresses to their jump boxes, I don't bother); to get to real hosts, you bounce through that one.
A network where you can randomly connect to the sshd on real servers is one most security professionals will assume isn't being run competently.
Ja, should have been more clear. Removed password login while maintaining existing keys.
Yeah I find young naive devops people implementing security watching and stuff like fail2ban but in the long run it reduces productivity. Stuff like fail2ban can block legitimate traffic (it blocked me on many occasion trying to work from home).
When I was young I too used to try implement detection and banning but realised it's a waste of time. Key only access to ssh and make sure you at least subscribe to security updates for all packages is enough.
For our own app we went the ELK route due to the number of log messages coming from our own application. But yes as you said it's a pain to manage. Also creating kibana graphs/views/desktop for each type of log is also painful.
A good service I believe would be someone to offer a turnkey/shrink wrapped managed ELK for self hosting. Not only managing the system side of it, but also offering services to help create custom kibana graphs.
Just set a short ban time in fail2ban. I have it set to 5 minutes. The aim is just to slow down the kiddies so that my logs don’t fill with a zillion pointless attempts.
And/or just reconnect through another server or VPN.
Yes, but my aim is more to create a molasses pit to slow down the log fill rate rather than to ban anyone for long. If I manage to lock myself out I just get up and get a drink of water.
Using a short ban time keeps the load of running fail2ban to a minimum. I at one point was banning the kiddies/bots for a week and 40% of the server load was chewed up just running fail2ban.
Fail2ban has issues when you have a lot of IPs blocked in memory.
The solution I came up with to this problem is to use a custom ban action that adds an IP to the blocked ipset list with a given TTL, but set fail2ban to think that the IP has to be only banned for 5 seconds or so.
The result is that ipset purges the IP from the list after the TTL expires, but fail2bans working set is a 5 second rolling windows of blocked IPs.
This works great and it's being used in production for a couple of years.
Sounds like a great solution. Have you written about this in any detail?
Do you really need the space or is it just some anxiety issue / ocd issue?
If a sufficiently interested botnet discovers your SSH server, it'll hammer your server with 10's of connections a second, from many different IPs. That doesn't take up enough bandwidth to be noticeable, but it can fill up the disk, when you're least expecting, which can cause all sorts of trouble.
Of course there are ways to deal with it, but that's just extra work.
Fill up a terabyte? Doubt it, especially since logs are compressed nightly.
Are you serious? There are others https://riskemy.com/free-cloud-vulnerability-scanning-tools/
What does your comment have to do with mine? You seriously registered an account just to reply to me? What's going on?
When your server is down and you need to ssh into an arbitrary 5 minute wait is going to frustrate you.
You don't really need fail2ban at all. Just don't log failed password attempts. Key access only, no one is going to get in.
I am not worried about people getting in via ssh, I am worried about some zero-day on some other service.
What about moving SSH to a different port?
I do this, but it is the port scanning that I am tracking. It is hard to find your ssh port if you get banned port scanning.
Removing attack surface make it viable to continuously monitor. I don’t think it is an either/or.
Just out of curiosity has anyone with a cryptographically strong ssh password ever been hacked via the password authentication path unless they gave this password to someone else?
I don't know what "cryptographically strong" means in the context of a password, but I do know people with strong SSH passwords that got owned up, yes. Don't use "cryptographically strong" passwords with SSH. Use keys, disable passwords.
I mean a password with enough entropy that also hasn't been used before and is not shared with anyone.
Do you know how they were owned? Is there a write up anywhere as this is a super interesting topic.
For passwords "cryptographically strong" normally means a password with appropriate complexity and sometimes unpredictability requirements, since entropy is almost nonexistent in passwords.
Something out of pwgen or diceware would do though. If your application has length requirements that preclude the latter, make sure they are liable if you loose something due to preventing you from using a secure password. (No one can remember a short secure Password for more than a small handful of applications.)
By having their password brute forced, or some other means?
So for example, a 16 character random password is still vulnerable?
This is what I would like to know. If strong ssh passwords can be broken then there is a bigger issue than just turn off ssh password log in.
I am pretty sure that 16 character random passwords are safe from dictionary attacks, and I suspect you and I are using the same process for protecting these passwords. tptacek is leary of our process while you and I are leary of the certs getting out of our control without our knowledge. Security is a handful of trade offs.
Good answer, I only came to add that OSQuery should be included in your tool box. Much easier to query for indicators of a problem, especially at scale, than it is to grep through logs and reconstruct an event. This is still a detective tool that would not likely catch an incident in real time, so the preventative measures listed are still best practice, but it should be included as a tool to fall back on.
Security monitoring is useful and part of a defense-in-depth strategy. So is fail2ban and many other things.
"Defense in depth" is just a way security product salespeople get customers to buy more stuff. If you remove attack surface altogether, you don't need more depth of defense.
"just" that? That's a very bold statement that most of the security community would disagree with.
An example: you don't monitor network traffic on the border because you trust your server not to be compromised. Then one gets compromised and the incident goes undetected due to lack of monitoring.
That's a big if. Services can have a complex life, with software and configuration changing often. Unless you pull the plug you attack surface is never going to be zero and, on top of that, there so guarantee that it stays small. Hence organizations usually have different people taking care of different layers.
I'm pretty happy with sumo. A couple things I wish they'd change, but their core competency is searching logs and metrics and they do that really well. It is paid (there is a free tier but we outgrew it quickly) bit it's well worth it IMHO.
So much this. Logs are great for telling you something bad has already happened and what that was, both security and faults. Reducing the attack service will get you more mileage from a security pov (again, in my opinion). Logs need to he there and some basic trawling of them is also necessary, but is purely reäctive.
Amen to that. I stopped caring about these sorts of entries in my HTTP logs when I realized these automated attacks target software packages written for language runtimes I don't even have installed. I'm sure not losing any sleep at night that someone is trying to hit phpMyAdmin or wp-admin on my server that doesn't even have a PHP interpreter.
A while ago I was looking for a very lightweight system to set up a small CMS with no bells & whistles (Pico, Grav etc), but all of them seem to require PHP. Are there any alternatives around which could also dodge most common attacks by not having installed most common interpreters?
Look into static site generators... you basically write your site offline, and run a script to deploy it to a static web server (or mostly static). In the end, a web server that doesn't run any code outside delivering content is the most secure web server. It'll be fast, and (usually) near zero attack surface.
this is essential, love the analogy here: https://youtu.be/NUXpz0Dni50?t=14m13s
previously worked at a major shared webhost a few years back (which used Apache)
we used mod_security with common rulesets + a custom list which was reviewed/updated based on what hacks were coming in (with hacks both passively monitored by customer base/compromised spam reports and via active scans for malware deposited by attackers which were then cross-correlated/traced back to application logs to find the entry point/compromise.
support staff was ~600 (including front end call center and billing people), with probably 250 of those doing L2+ support and ~50 doing the security work which fed into the ruleset, for a server farm of ~2-4000ish (and exponentially more sites/accounts since it was shared webhosting). In some ways our network could have been viewed as a giant honeypot because of all of the outdated webapps that were being run by the customers, plus their compromised PC's saving passwords, etc.
I see a lot of mod_security misconfiguration across hostings. While I think it is a great tool, these misconfigs make it difficult for me to recommend it.
There's really no way to eyeball logs in any sort of complex environment. What you're looking for is a system that will automatically analyze logs and/or traffic for malicious events and alert on them. What you're looking for are Introsion Detection Systems (IDS) and Security Information and Event Management (SIEM). For IDS there are a whole bunch of open source solutions. Essentially these systems ingest logs into a centralized location and run them through a rules engine, which gets updated with new rules on a weekly or daily basis.
(I don't pick on you, so I hope this comment doesn't come off that way.)
What are "malicious events" and how will you know them when you see them?
I would argue that, instead, your system should automatically analyse logs for benign (a.k.a., completely normal, expected, innocent) events, toss them out, and leave humans to review anything that's left over.
With any new "system", it will take a bit to build up a complete "whitelist" so that the amount of data to review is down to a manageable level but you'll be rewarded with much more meaningful and actionable data when you get there.
Instead of the term "malicious events" he should have used the term "anomalies".
I suspect they are simply defined as "rare" events.
I work in a Fortune 100 Security Operations Center (SOC). We have a dev team that builds tools for our Incident Response team, Threat Intel team, etc, etc and an engineering team to manage those tools. And lots of other teams that specialize on lots of other things (malware reverse engineering, spam/phishing, red team, etc). We have to build a fair amount of our own stuff because we operate at levels above what a lot of commercial tools can do. We have hundreds and hundreds of custom alerts that fire based on various network traffic and sysmon activity that our threat team has written rules around.
This, of course, is out of the question for most.
So, what should you do? First, get your logs all in one place. The ELK stack is probably the easiest way to do that. ES makes things searchable and Kibana is a pretty good UI for a lot of this stuff.
Second, if you aren't monitoring your internal network, you really have no idea what is going on, so look at what you can accomplish quickly with distributions like Security Onion (which includes the ELK stack): https://securityonion.net/
Third, there's a fair amount of Open Source Intel (OSINT) on threats. You'll need to find a way to integrate that information and scan your logs for it. Malware, like Coinminers, will have certain Indicators that will tell you if they are on your network. If you're on the ELK platform, look at something like ElastAlert. You write rules in Yaml and then set up Alerts to fire into your Slack channel or Email you or whatever. There is no real shortcut here.
Commercial tools that can do a lot of this stuff includes FireEye's TAP, but they definitely aren't free: https://www.fireeye.com/solutions/threat-analytics-platform....
And, as mentioned elsewhere, absolutely remove any attack surface that you can. Scan your IP ranges and domains with tools like Shodan.io, HackerTarget.com's DNS tools (DNSDumpster.com has a nice, free DNS search), etc and make sure you have a handle on what, exactly, you're exposing.
And, finally, if you're big enough then every single day a computer will be compromised in some way. There is no company in the world who doesn't experience this so you need to have a clear set of tools and playbooks to handle that. If you can respond to most incidents in days (not weeks), you'll be better than 90%+ of the companies out there.
Edit: clarified acronym
"SoC"?
Sorry, Security Operations Center: https://en.wikipedia.org/wiki/Security_operations_center
Im currently building a SoC for a fairly large company in Germany and before that I worked for what probably by now through M&A is the largest MSSP in the world. I dont know if I agree with you that having a MSSP is out of the question. we had plenty of clients who only paid us maybe 2 to 3 thousand a year if even that. Most MSSPs will charge on a contract to contract basis revolving around log volume and estimated daily event creation. If you are only creating an estimated .1 events per day you could very well have a rather cheap contract.
If it costs 2 or 3k a year your MSSP is probably not spending more than a few minutes a year understanding your logs. Realistically at that price the most likely number is zero minutes.
There is room in the industry for players that aggregate logs in interesting ways and can share threat intel and track attack patterns between customers, but a lot of the time with MSSPs you will just find yourself paying for a really expensive splunk instance.
Logs are fairly useless without occasional review (to at least verify you are collecting the things you think you are) and a decent understanding of the infrastructure they support.
I don't know your business model or capabilities though so this might not apply in your case. Hopefully not.
But there are plenty of MSSPs out there that will ingest logs, charge out the %]%{^{ for "monitoring" and add very little value. After the nth time you get alerts for PHP vulns on stuff which is clearly not running it, or something stopped sending logs months ago and no one noticed, or you find that firewalls logs from anything not in an unpublished support list don't really get parsed (yeah they will ingest it but just not tell you that nothing meaningful will happen with it..)...
To be fair, monitoring by third parties is one of those things that you can't "do and forget about it", you need shared responsibility and a clear understanding of who is doing what. None of this is cheap though.
SpectX is a tool that can run fast sql-like queries on top of distributed text files. It was initially built to parse logs by an infosec team. Hope it helps!
For me monitoring the logs is already too late to do anything about the hacking attempt. Usually web servers write logs after the request was served. One approach (I have seen commercial products do it) is to hook into the web server code and inspect every request real time before it hits the application servers and make a decision if it is a legitimate request or not.
It does take a number of malicious requests before someone can find an oppening. If you ban them after the 1st or 2nd request, you will be safe. It is going to be a miracle if some vulnerability scanner finds and oppening after 1 request, not impossible but HIGHLY unlikely
Sadly, it is too easy to get a new IP address now. I think as part of our security mindset, we should consider it trivial to scan for several hundred vulnerabilities using relatively unique IP addresses.
I'm not saying that you shouldn't ban traffic as it's required under several compliance programs and it is another form of demonstrated diligence. But, unfortunately, banning individual IP addresses only works against the absolute lowest hanging fruit of the criminal world.
A tad off-topic, but one thing I've enjoyed doing in the past is giving my attackers something to get distracted on, allowing me to observe their behavior.
https://www.youtube.com/watch?v=smvwW3vW5rM
Wow, it's interesting to see how bad this hacker is with a shell. Even the "better" attacker from the video you linked in the comments doesn't seem to really know what he is doing at times.
Sadly, I feel not all hackers are dilettantes.
Another question- how do you comply with GDPR and do that? What if a black hat sends you a SAR request?
I use logwatch. It gives a large informative report every day. Mostly it shows me that fail2ban and my secure configuration are doing what they are supposed to do.
I sleep a lot better since I use IP white lists on any port/is that is not public facing. Central reliable location with a signed whitelist that the servers checks regularly and adds to its firewall. No security measure is bulletproof, particularly against zero days but that reduces the risk by orders of magnitude.
Nice idea, but how does it improve over "have important stuff on an administrative VPN"? To me the whitelist seems more effort (but we often have non-static IPs at home; with dual stack lite aka native(?) ipv6 plus a private ipv4 behind a huge, ISP-wide NAT making things even worse), and it doesn't give you additional encryption.
Agree. I have scripts regularly updating the whitelist with dynamic IPs every 5 min. But I don't use VPNs for various reasons.
OpenVPN on windows is very slow (unlike on Linux), it's hard to get above 5-7MB/s, which is a problem for large file transfers (and that was pre meltdown and spectre patches). It's also hard to configure, and the server still has WAN connectivity so it's easy to misconfigure it and get no protection. I also found that having an OpenVPN LAN with no gateway in addition to your default LAN on win7 tends to interfere with Chrome on the client, which will occasionally complain that it has no connectivity.
Being a belt and suspenders guy, and only dealing with windows machines, I found that a good solution is to combine it with IPSec, which is straightforward to configure on windows, fast, and adds encryption when the underlying protocol doesn't have any (SMB2). You can configure the firewall to only use IPSec for certain ports. But it's a nightmare to interop with non windows machines.
It's called SIEM, Security Information Event Management, there is a lot of companies doing this and providing cheap and expensive solutions.
For web logs specifically, even if you had a system that would 100% tell you what is a hack attempt and what isn't, you would probably not want to look at the output.
The web server will be hit by thousands of "hack attempts" a day. Most of them will be directed at software you don't even use.
You care about successful hacks, not random attempts (targeted attempts may be different, but good luck finding them in the noise of web logs). Web logs are probably the noisiest, least useful place to look for those (although they can be useful to determine how an attacker got in/where they came from after the fact).
The only thing that could make sense for web logs is looking for brute-force attempts, but that is better done with application-level logging (and countermeasures like CAPTCHAs).
Things you could look at instead are e.g. firewall/netflow logs: If your server is making unexpected outbound connections, there's a good chance it got owned and someone is now downloading their toolkit onto it.
Don't.
You shouldn't care about hack attempts. There will be thousands each day. They don't matter.
You should care about your systems not being easy to hack. Unless you're a very high value target then "practice basic security hygiene, install your updates, don't reuse passwords" is enough to achieve that.
There is a product called Unomaly [1] which does exactly what you wrote.
1. https://unomaly.com/
Ha I remember awhile ago doing something similar with our AWS server logs, turned out we were always under attack 24/7, rather amusing but not very useful apart from scaring juniors a little bit.
I can not recommend watching this Defcon talk enough: https://www.youtube.com/watch?v=_SsUeWYoO1Y - "You Spent All That Money"
Dude I'm a security consultant for a major vendor and I can't tell you the number of times I've
A) Found an incident while installing new equipment
B) Installed new equipment last year and came back this year because they still got hacked
and/or
C) Ripped out their old gear and put in my company's gear after a hack because the sales guy told them they wouldn't get hacked with our (passive, monitor-only) gear.
You need the right tools, no doubt. But you need to understand your needs first. Centralized logging is the first step, but without knowing what you want to get out of it, those logs just sit there unnoticed. Even intelligence platforms like Splunk or ArcSight, unless you know what you're looking for and write it into rules, it's just sitting there.
Knowledge is what matters, the tools just make it easier.
The most important things for this are: - A good idea of what is normal traffic and behaviour on the web service. - Good threat intel.
These 2 combined will make it fairly obvious what needs to be done, but obtaining these 2 items is not a trivial task, infact, they are both continuous processes.
Marcus J. Ranum's "Artificial Ignorance" concept is pretty good. I did an implementation and have been running it for more than 10 years:
http://www.ranum.com/security/computer_security/papers/ai/
Someone's also done a Python implementation: https://github.com/lchamon/ai
...and it looks like there's one in syslog-ng Premium Edition:
https://syslog-ng.com/documents/html/syslog-ng-pe-latest-gui...
I used to run a web service, an API, that created 3D models of people from photographs. It was a character creation service that made likeness models of real people from a photo. I knew we'd get a lot of hack attempts, and a consultant recommended we get a high end Sonic Wall appliance, a $20K piece of hardware that cleanses all network traffic to and from my data center.
Best decision EVER! We'd get hack attempts, we'd get DoS attacks to the tune of 130K hits per hour, and that Sonic Wall shrugged it all oof like nothing. For a while, it looked like every script kiddie was trying to hack into our servers. No one got through, and we could care less how or what they were doing in their attempts. We had real work to do, that did not include fucking around with those idiots.
We also had a block list for customers from China. Not a single transaction from China was legitimate, all forged credit card data, so we just blocked their billions of potential customers because not a single purchase from China was real.
We use a combination of https://www.templarbit.com and Splunk. Templarbit allows us to get real-time notifications when someone is running a vuln scan, violates one of our security headers or is trying payloads that look funny.
Could something like poor man's primitive check work?
- Create a custom http header and find a unique way to create/identify it. So theoretically another layer of authorization, just like APP ID for instance.
- Create a middleware that would scan the request and check whether it has the required custom header
- If not log everything and create a notification/slack/allow/deny and if it's good let through the request.
Separating this out:
1. Monitoring logs for security -- people don't, but machines do. Initial security comes more from segmentation architectures (bastions, whitelisting, ..) and network/host intrusion detection systems. Logs do come into play later for app-specific rules (unpatched apps, fraud tipoffs, ...) and eventually automated UEBAs (exabeam, ..) and analyst threat hunting (plug: http://www.graphistry.com !).
2. Forensics for post-breach attribution & containment/mitigation. Let's say something does happen (... it will.) Logs are important here.
-- Threat intel: Known indicators of compromise (IOCs) -- IPs, URLs, downloaded file hashes, ... -- get published. Team setup always-on correlation rules against their internal logs via say splunk or hadoop, and if hit, they know the threat intel of it. Or they'll get alerts from OpenDNS that an internal device is talking to a known bad actor. That's generally pretty shallow threat intel on the attack (ex: VirusTotal.com entries), but a start. There are multi-step ways to iteratively correlate across these sources, but takes effort. Attacker attribution happens here, but typically more important is identifying further incident IOCs in case of a bigger breach.
-- IR / DFIR: Teams care more about defense and explaining what happened internally (incident killchain, scope, ...). Say there was a malware alert from AV, if the dwell time was hours or the device important, teams should investigate what else happened for the device/user/etc, or if anyone else got hit. THAT'S where logs really shine. Small/immature teams will check individual tools (firewall device, ...), while more mature teams check centralized logs.
We find manual log analysis shallow in practice for a decently sized org... so another plug for http://www.graphistry.com for setting up more comprehensive multi-step analysis flows for the human side of looking through stuff like malware, phishing, and account hijacking. Tools like ours work on top of log stores like Splunk/ELK/Hadoop/etc. (more scalable) or bring their own DB (less scalable.) This can get pretty intense: some companies log all the way through process activity.
At the extreme end (FBI, Mandiant, ...), companies will quarantine a device for activities like analyzing RAM & disk images.
ossec + syslog , how you display the data (zabbix, elk, etc) is up to you.
I'd recommend only really addressing the most obvious attacks, rather than trying to follow up on everything that even looks like it.
I once got an "emergency" call at 7am (I had been up doing something rather late), about a machine I run attacking this guy. After some digging with him, the machine was our public-service NTP server, and the packets were on udp/123, in response to packets his system was sending out. Ended up with him yelling at me and hanging up, and me shutting down our public NTP server for good.
What you need is information, what's easy to get is data. Generally, if you have data on what "normal" looks like, and data on what the current state is, you can view the difference as information that indicate an abnormal state; be that a full/failing disk - or someone downloading all tour data via blind sql injection.
That's the theory. I've yet to achieve this in a timely fashion, within budget limitations - but on my list is to have another look at graylog2 for logging (if you have the money, splunk is probably easier - but they do charge for the value they provide).
But I'm hoping to find the time to mash graylog2 with something like collectd, and maybe a general event collector for exceptions/stack traces. The goal would be to have it all on a similar timeliness in a unified interface (this spike of exceptions happened after this series of requests from there, and after the disk went full). And ideally with a sensible way to automate "normal" vs "exceptional" (alert on high cpu, low disk, high network bandwidth etc).
Sure, people should read logs/alerts - but it's a challenge to filter out irrelevant (ssh pw bruteforce against a ssh server with only key auth, Web app crashing due to #wontfix (yet) bug) - so the relevant becomes obvious (successful key based login from different continent - where none of your team are).
https://github.com/Graylog2/graylog2-server
I highly recommend you try out Sqreen. https://www.sqreen.io/ It's basically newrelic for security. Takes 2m to add to your web app and it works super well. It blocked an attack automatically on our service recently!
error/exception monitoring
Dont do this in production without backup tools but we use unsupervised learning for anomaly detection. We're machine learnists so this is natural for us and we originally built the system for a mega Corp and made it open source (though not all parts are yet public). Catch is it's not real time but he system outputs outliers every few hours or daily depending of the size of your traffic (daily for around 50M proxy daily logs). Outliers are then reviewed by scripts and few operators / analysts.
I work for: https://www.signalsciences.com/ and our tool is specifically designed for use cases like this. Let me know if you would like more info cody-at-signalsciences-dot-com
WAFs like CloudFlare help, but stuff still gets through. Last night an obvious brute force attack went undetected through CloudFlare. I just finished writing a post about it: https://hauptj.com/2018/05/08/brute-force-login-attacks/
This is a list of what you can do for application security with Nginx (mostly with open source tools): https://github.com/wallarm/awesome-nginx-security
My talk from Nginx conference: https://www.nginx.com/blog/build-application-security-shield...
Important note. Care about vulnerabilities. Not about attacks. Buy Burp license. Run appsec training for all of your developers; it's easy while you're small.
Disclaimer: I am a co-founder of Wallarm mentioned in preso.
I use Fail2ban http://www.fail2ban.org/wiki/index.php/Main_Page for auto blocking.
I take an 80/20 type approach and see what types of access is not covered by current rules, then I add a custom rule to cover that type of access attempt.
During my time at Wanderu we set up Filebeat on all app machines, forwarding json logs to Kafka, and then on to ELK and Pipelinedb for analysis/alerting/session killing/ etc.
It worked very well, as there are almost always application edge cases that should not be blocked immediately with Cloudflare or at a FW level.
ELK was used for exploration and/or setting alerts on thresholds that we had found/defined beforehand.
Pipelinedb was used to run continuous aggregates on a stream, augment (pop json log lines off a stream, enrich them with Maxmind, etc) logs in "realtime", and immediately surface malicious behavior. We'd then programmatically kill sessions or add Cloudflare rules upstream, based on the behavior that surfaced in a Pipelinedb continuous aggregate.
Wanderu: https://www.wanderu.com/ Filebeat: https://www.elastic.co/products/beats/filebeat PipelinedB: https://www.pipelinedb.com/
I'm curious how you got Filebeat to push to Kafka. Did you write a custom modification to Filebeat/libbeat?
I maybe have a counter-question. Once you identify an attack, what do you do? Firewalling the attacker is an option but not really effective, the attacker might just change IP address and keep going. We've only inconvenienced them a bit. What else can be done?