Are Automated Scans Enough to Detect All Security Problems in an Application?

Automated Scanners Are Great Tools, But Are They Enough?

Spoiler alert: No, automated scanners alone cannot cover all aspects of a holistic application security plan. However, I suspect more details are in order, so I can’t end it here.

For this post, we’re really talking about two main types of automated scanners: Dynamic Analysis Scanning Testing (DAST) scanners and Static Analysis Scanning Testing (SAST) scanners. As implied in the name, DAST scanners run against an application that is running, whereas SAST scanners run against an application’s source code. We’re also going to take a look at the Payment Card Industry (PCI) scanner sub-category (generally these are DAST scanners, but SAST scanners will typically have a PCI setting as well).

So why are there two types? In short, each has its own set of strengths and weaknesses. For example, a SAST scanner can find hard-coded passwords and unencoded outputs incredibly easily; it’s looking directly at the source code, after all. Since a DAST scanner works so differently, it has a harder time finding those glaring source code issues. However, it is a bit easier for a DAST scanner to check for other issues, such as authorization. Not nearly as well as a human can, but still better than most SAST scanners. DAST scanners have one other benefits, including the fact that they can check web server configuration. This involves checking for things like default web server pages, fingerprinting, or directory browsing. In other words, each scanner brings something to the table, and by taking advantage of the strengths of both, a reasonably thorough application assessment can be performed.

Obviously, these tools are really handy. They hook into the Software Development Lifecycle (SDLC) well and allow for quick turnarounds on assessments. Hit go and you’re probably only a few hours or less away from the results. Additionally, by tackling all the low-hanging fruit (like the aforementioned hard-coded passwords), your security team is able to spend their time on more difficult issues.

So now for the real question:

If they’re this great, why are automated scanners not enough?

Scanners May Miss Authentication and Authorization

I touched on it earlier, but automated scanners are far from amazing when it comes to checking for authentication and authorization issues (Make sure to check out our blog on Authentication vs. Authorization). They can do it, but for the time being, humans win by a long-shot on this task. These tasks contain some nuance that are hard to tell a scanner how to perform without spending massive amounts of time on configuration. Obviously, this would get messy, if it’s even possible to configure reliably at all. A human tester has the benefit of being able to “feel” (for lack of a better word) when they can access data or functionality that they shouldn’t be accessing.

Automated Scanners Can’t Validate or Check a Logic Flow

Arguably the most glaring deficiency of automated scanners is their inability to check business logic. While some of this dips its toes into authentication and authorization, it goes a bit beyond that. Let’s take an example of a web store, is it possible for a user to skip the payment step and go straight to the shipping step? This is another example of a vulnerability that is scanners miss but would be well within the ability range of a human tester.

What is a PCI Scanner?

A PCI scanner is an automated security scanner that looks for issues that would cause an application to not meet data security standards set in the PCI Data Security Standard (PCI DSS). Currently, PCI mandates that PCI Scanners must identify OWASP Top 10 vulnerabilities.  Many PCI scanners run against a running application (essentially a DAST scan), so it will generally face the same obstacles.

Is Using a PCI Scanner Enough? 

In short, no, and in this case, there’s a little bit extra to consider. Not only does a PCI scanner have the all the normal downsides of an automated scanner, it also has an explicit focus on checking PCI compliance. While this might sound fine initially, PCI compliance does not mean that your app is secure, it just means that PCI says you performed one step towards allowing you to handle credit card payments. In other words, a PCI scanner has a narrower focus.

If The PCI Scanner Gives the All Clear is That Enough?

A PCI scanner giving the all clear is a good thing, but enough, it is not. As I briefly mentioned earlier, a PCI scanner has a clear goal: to check if your application has identifiable vulnerabilities in the OWASP Top 10. Since this goal is not the same as broadly checking the security of an application, additional measures should be taken.

If the site is PCI Compliant from a network perspective, does that also mean it is secure?

No (big shock, right?). PCI scanners check for network issues in the same context as application issues. In other words, checking to see that PCI standards are met. This once again means that issues beyond the scope of the PCI standards may not be checked.

How Do You Define Enough?

There is no way to be completely secure; it’s an ideal that is not practically obtainable. However, there are ton of ways out there to reduce the risk of breaches. Furthermore, I would argue that automated scanners are an effective use of resources to reduce security risks. Although, they still lack certain qualities that human testers possess. In other words, they alone are not “enough” (at least not in my book). An ideal approach to security combines the two, taking advantage automated scanners to find as many issues as possible, and using humans to clean up the remaining, harder to find vulnerabilities. This works even more effectively in the context of the Secure SDLC, where the goal is to get as many security issues found as early in the development process as possible.

Ultimately, it’s up to the individual or organization to determine how many resources they want to devote to security, and what level of risk they are willing to accept. This is by far the hardest question, and also not one that I can answer for anyone else.


About The Author

Cypress Data Defense

No Comments

Leave a Reply