Lyal Collins is a Loop PCI-DSS expert and he advises Loop clients on compliance with the PCI standard. In this post, Lyal talks about the PCI council’s Guidance Paper on Penetration Testing. If you have any questions for Lyal, please email info@looptech.com.au
The Payment Card Industry Data security Standard (aka PCI DSS) prescribes three active testing controls that require specialised tools or services as opposed to other controls that can leverage common tools used in most IT environments. The three active testing controls are:
- Penetration testing
- Wireless scanning
- Vulnerability scanning
While wireless scanning and vulnerability scanning are less expensive to conduct and produce highly reproducible results, the results of penetration testing (pen testing) are not as easily reproducible. This shortcoming of pen testing, prevalent in less skilled or less professional provider of these services, invariably leads to a lack of complete assurance that there are no easily exploitable weaknesses in the systems being tested. This is not a happy situation for businesses that invest dollars insecurity compliance.
As an experienced PCI-DSS consultant with over years as a QSA, I have been acutely aware of this problem with pen testing for a number of years and therefore our company has evolved a set of guidelines that address the root causes of this problem. Contributing to our internal improvement process has been our QA process, and our separate PCI Qualified Security Assessor (QSA) practice, which requires Loop’s QSA staff to assess the reports arising from pen testing from many pen testing firms, not just our own team. As a result, we have access to an ongoing metric of how we compare to others. Therefore our pen testing methodology has consistently provided highly reproducible results and a high degree of assurance to our clients.
Sadly that is not the case with majority of the pen tests carried out around the industry.
As a PCI QSA, reading pen test reports is a regular component of my engagements. It is painful to both me and my client when I tell them that they have paid for an expensive pen test that was little more than a vulnerability scan and that didn’t give me any confidence that the client was meeting the intent of the PCI requirements. In some cases, the report wasn’t worth the paper it was written on – and not just because the report was delivered electronically!
Effectively, the client is faced with failing their annual PCI assessment, and having to pay again for a new pen test, or delay their PCI Report on Compliance and face financial penalties from the card brands.
When it happens, this lack of ethical and professional behaviour erodes client confidence in pen testing, creates resentment in what is seen as a “tick in a box” process, and reduces available resources for other security processes and outcomes to mitigate risks present in every organisation.
Given this background I am very glad to see that the PCI Security Standards Council has released a guidance paper, providing a lot of useful details and pointers for procuring and delivering pen testing services against the PCI DSS requirements. As such, it is aimed to help organisations that buy pen test services, and also sets aspirational goals for pen test teams to achieve or exceed, every time they are engaged to deliver pen tests. The guidance paper is based on input from industry experts involved through the PCI Council’s Special Interest Group process, and thus is based on real-world experience and outcomes from the tester and target perspective, rather than abstract principles.
Available from https://www.pcisecuritystandards.org/documents/Penetration_Testing_Guidance_March_2015.pdf, this guidance paper outlines a framework with the key goal of increasing the quality and assurance to be obtained from undertaking pen testing. The guidance paper is a good reference for anyone procuring or delivering pen testing services.
The paper addresses the main reasons that lead to the uncertainty and variable assurance or confidence in the results of pen testing:
- Human factors that can introduce variability into the testing process – energy levels, mood, skills and experience.
- The (potentially) millions of testing permutations to perform and results to analyse across today’s complex web applications and infrastructure.
- The specialised skills required to analyse the results of every individual test in order to confirm if further investigation is needed into the target system or environment.
- The difficulty of scoping the test for the greatest coverage while in order to deal with constraints like budget and project release dates.
Regardless of whom our PCI DSS clients choose for their pen testing needs, I recommend the PCI Council’s guidance paper as a great step forward in improving reproducibility and increasing assurance in the usefulness and benefits of actively testing the effectiveness of an organisation’s security controls.
