An oped on the issues of CISSP

techfiend

CISSP certification: Are multiple choice tests the best way to hire infosec pros? | Ars TechnicaI'm not in security but I agree with it, especially if CISSP has questions like the pic in the article.I still may pursue CISSP since it's gold on a resume. It appears to give sufficient knowledge for management which is my focus. For technical roles I'm unsure.

Danielm7

The article is a mixed bag. The problem with the CISSP is that a lot of people feel it makes you a security guru in all aspects, which is totally untrue. They compare it to the OSCP, which is like apples and oranges even though they both are in the realm of security. There are enough specialties in security that looping it all together is like saying you can get one cert and it makes you the master of networking, servers/systems, voice and programming, which is also impossible.

As for the multiple choice part, I think for the CISSP it makes sense. It is largely management, DR/BCP, etc. I'm not sure how you'd answer those sort of questions in a hands on manner. For highly technical things, it makes sense to be able to demonstrate.

aftereffector

Daniel is correct. The article is not technically incorrect - the CISSP is often misused as a measure of technical expertise - but ultimately misses the point. No one seeking to hire a pen tester should put much weight behind the CISSP, the CEH, or any other knowledge-based exam. Skills demonstration such as the OSCP or capture-the-flag are far better suited to determining whether a candidate is competent at penetration testing. However - and this is the point that the article's author misses - the CISSP is not focused at pen testing, and in fact the infosec field is much broader than pen testing anyways.

g33k3r

I agree with the comments above. In my opinion NO certification is perfect. It is a starting point just like any other knowledge based achievement. It is what you do with it after it is obtained (certification or college degree) that is important. In a general sense it tells me you spent time (usually outside of work) learning something focused on a specific subject. The content may not be completely up to date, but consider how many technical books are outdated within a year. During my career as a Sys Admin, my lenses were focused primarily on technical solutions to technical problems. This carried over to how i viewed security. What the CISSP taught me, which carried over into my security career, was how to integrate security with the goals of the business. GRC has helped me gain support from senior leadership so that new programs and resources are prioritized and become successful.

I worked very hard for my CISSP, but that hard work didn't stop after the exam. A day or two later I was surveying my skill set to identify my areas of weakness and what my job required. From there I've developed a plan to LEARN material relevant for my job. Fortunately my job allows me a wide breadth of knowledge which is both good and bad. Some of the things I've focused on recently are:
- NIST 800-xxx
- FISMA
- Risk Management
- Policy and Procedure development
- Web Application Security
- Python
- Offensive Security Techniques

I probably left something out, but this is what my employer needed me to do for our organization. I don't ever foresee myself as a penetration tester, but that doesn't mean I shouldn't be somewhat familiar with techniques and skills required. I purchased the PTSv3 & PTPv4 training which has been excellent. My only complaint (sarcastically speaking) is that it has lead me down a rabbit hole to pursue a greater understanding of some of the areas of study.

I found it interesting that the author is/was currently studying for the CISSP but is now reconsidering his decision. Before I bought my first CISSP study guide, I had a basic understanding of what the exam involved. I understood that it was not an exam designed to gauge penetration testers.
Sorry for the long winded opinion.

gespenstern

The Ptacek guy is a former appsec pentester (at least he claims that he is) and now he's a recruiter. He has a long history of sh!tting on CISSP from a pentester viewpoint, thus OSCP instead of CISSP. Nowadays he seems to be pretty busy tweeting tens of tweets a day on political issues, such as demands to disarm majority of police force, sh!tting on Trump and protecting Hillary in email scandal.

g33k3r

gespenstern, good to know!

ITHokie

Article wrote:

"I push back on the idea that there is not enough talent out there," he says. "We don't need to train a new generation; we need to do a better job of breaking down the wall that HR and tech managers put up as an excuse to not bring people in."

Yep, nailed it. If you're management track or policy maker track, I do think it's somewhat useful. If you're going technical track it's (sadly) great for marketing yourself but woeful for learning/validating technical skills.

http://www.techexams.net/forums/isc-sscp-cissp/105191-passed-cissp-10-24-disappointing-expected.html

ITHokie

gespenstern wrote: »

The Ptacek guy is a former appsec pentester (at least he claims that he is) and now he's a recruiter. He has a long history of sh!tting on CISSP from a pentester viewpoint, thus OSCP instead of CISSP. Nowadays he seems to be pretty busy tweeting tens of tweets a day on political issues, such as demands to disarm majority of police force, sh!tting on Trump and protecting Hillary in email scandal.

Way to poison that well.

Danielm7

gespenstern wrote: »

The Ptacek guy is a former appsec pentester (at least he claims that he is) and now he's a recruiter. He has a long history of sh!tting on CISSP from a pentester viewpoint, thus OSCP instead of CISSP. Nowadays he seems to be pretty busy tweeting tens of tweets a day on political issues, such as demands to disarm majority of police force, sh!tting on Trump and protecting Hillary in email scandal.

Reminder to always consider the source.

Ritual

The article brings up a good point though.

I just started in IT certifications and I would say 80% of the people I met so far were using braindumps.

Practical exams like Ofeensive security, RedHat, eLearnSecurity, Linux FOundations are a no brainer approach to how you should structure your exams.

Why doesn't the CISSP give you a scenario where you walk through auditing a system, writing a report, explaining compliance, etc etc. Wouldn't that bring it more prestige and better test the knowledge?

The problem I have is it seems like these certification vendors are making BUTLOADS of money from IT certification, and if only 40% of people could pass, they would lose out on revenue. And without lots of people having their certifications, it makes them less popular. Which means employers are less likely to put the certification as a requirement.

Its a damned if you do, damned if you don't approach to how you structure your exam.

I am going for practical certifications as much as possible because I think they would make me better. I don't care as much what an employer thinks about it as I do how I feel it helps me personally. I am doing the Linux Foundations Sys Admin, eJPT, eCPPT, and then one day the OSCP.

If you don't learn by doing, you aren't really learning.

I could of passed my CompTIA certifications without ever touching a computer.

goatama

So for the first part of that article I kinda wanted to slap the author and some of the people he was quoting. The whole point of the CISSP is that you have to have five years of experience *in* infosec in order to get the cert. You also have to have someone vouch for your experience. If it's discovered that you lied, you get banned from the cert for life *and* so does the person who vouched for you. Do people lie? Sure. But the system seems to work fairly well.

However, then I read page two. And I realized that that's exactly what we're doing where I work with our technical exam before people get hired. I also realized that we implemented that exam *specifically* because we hired someone with a CISSP that was a complete tool.

Remedymp

I could of passed my CompTIA certifications without ever touching a computer

Because it is based on comprehension. Not expertise. "Do I know XYZ" is the method of the exam.

Ertaz

goatama wrote: »

I also realized that we implemented that exam *specifically* because we hired someone with a CISSP that was a complete tool.

LOL. We have one of those as well. Has a master's degree that he must have plagiarized to get. Certs are a just a segment of a person's qualifications. They are a measuring point, to a limited degree, not a defining centerpiece.

ZzBloopzZ

Ritual wrote: »

The article brings up a good point though.

I just started in IT certifications and I would say 80% of the people I met so far were using braindumps.

There are NO proper brain **** for the CISSP in particular though. I just look at certifications as a standard, that a particular person at least "some what" has a certain level of understanding. Plus you must have certain certs for certain jobs such as for the DOD 8570.01. It does not matter how much you actually know, you have to have certain certs for those particular gov't jobs. However, I think this is more vital for the DC area in particular.

Also, a family friend is incredibly hard working and he was in security at a large company for many years. However, no one took him seriously until he obtained the CISSP. Then they started putting him on more critical/important projects. Next thing you know he just became a CISO recently and the CISSP is his highest cert he has.

iBrokeIT

Group A loves to sh!t on certification X because Group B loves to put it on a pedestal and make it into something more than it was intended.

Change the variables and presto! New article for next month.

ITHokie

ZzBloopzZ wrote: »

Also, a family friend is incredibly hard working and he was in security at a large company for many years. However, no one took him seriously until he obtained the CISSP.

Perfect case study. This, in a nutshell, is the issue with CISSP. It's HR/Management/Leadership perception that is the problem (not the cert itself). It's one thing to hold someone back in a professional services/consulting/contract role where you're selling credentials to customers. But doing that internally when you can directly obverse their skills and abilities? Simply asinine. It's truly a wonder that places like this have a functioning security shop at all.

gespenstern

ITHokie wrote: »

Perfect case study. This, in a nutshell, is the issue with CISSP. It's HR/Management/Leadership perception that is the problem (not the cert itself).

With the CISSP? That's a problem with any cert, including the OSCP, that is being put by this article author on the same place the CISSP is holding.

ITHokie

gespenstern wrote: »

With the CISSP? That's a problem with any cert, including the OSCP, that is being put by this article author on the same place the CISSP is holding.

Sure. And the problem with any employer is that they just give enough time off. The problem with any economic system is they're not efficient. The problem with any politician is that they are dishonest.

These are true statements but aren't really useful because they completely ignore the degree of variance in their respective domains. For example, some employers give no time off, some give a month and a sabbatical after a few years. It's the same with certs.

I do technical screenings for consultants for everything from security engineering to analysis to penetration testing to security architecture. Whether or not someone has CISSP has little bearing on on their technical knowledge and skills. Plenty of skilled people have it, plenty of skilled people don't. It does not seem to be an indicator of technical ability. It's hit or miss. If they have OCSP or CCIE they have skills. I'm not arguing that makes them the greatest employee, but they've got knowledge and skills I can work with.

In my experience, there is literally no comparison between CISSP and other major certifications in terms of how overvalued it is by the powers that be. That's also borne out by my own experience taking the exam. Complete waste of 2.5 weeks of study.

OctalDump

But who is using CISSP as their sole selection criteria? Given that CISSP implicitly means that the candidate has 5 years of experience, I'm sure that whoever is hiring would be interested in that as well.

The article as a whole is rather unsatisfactory. CISSP isn't meant to be proof of specific technical competence or the ability to perform penetration testing or incident handling. It is basically a generalist Info Sec certification, albeit relatively more in depth.

ramrunner800

OctalDump wrote: »

But who is using CISSP as their sole selection criteria? Given that CISSP implicitly means that the candidate has 5 years of experience, I'm sure that whoever is hiring would be interested in that as well.

The article as a whole is rather unsatisfactory. CISSP isn't meant to be proof of specific technical competence or the ability to perform penetration testing or incident handling. It is basically a generalist Info Sec certification, albeit relatively more in depth.

Your point might be valid if it was used in hiring as you describe, but unfortunately it isn't. It only takes a few minutes of perusing jobs to see that there are plenty of companies who require CISSP for any security position whatsoever. It is not uncommon to see jobs requesting two years of experience, but a CISSP is required. I've heard plenty of people advised to get their CISSP in order to make a career transition to IT.

The article was pretty spot on in, at least from my perspective working in infosec in the States. CISSP is on the decline here, though that's probably a good thing as it is tremendously overvalued to start with.

OctalDump

ramrunner800 wrote: »

Your point might be valid if it was used in hiring as you describe, but unfortunately it isn't. It only takes a few minutes of perusing jobs to see that there are plenty of companies who require CISSP for any security position whatsoever. It is not uncommon to see jobs requesting two years of experience, but a CISSP is required. I've heard plenty of people advised to get their CISSP in order to make a career transition to IT.

But that isn't as the sole criteria, that's just another on a list of requirements. It's not that different to requiring x years of experience or a degree. And if you are hiring into an Info Sec role, then you want some method of describing a baseline. I think it probably does a reasonable job at that, in 5 characters or less. I also understand in some cases it fulfils an assurance role for the company, to demonstrate that they are taking reasonable steps to ensure qualified staff (industry standard is a usually a pretty good defence, even when the standard is not great).

I do think that work tests make some sense, but to represent it as a polar opposite of multichoice is not really accurate either. Any testing will necessarily be artificial since it needs to be a standardised (repeatable) environment with clear measures to be able to fairly and meaningfully assess and differentiate between candidates.

And there probably is a case to be made that a test administered to 10,000s might do a better job than the test made up by some manager with no real HR or testing training could do.

Personally, I'd much rather have organisations get to grips with what the actual skills/knowledge a job requires, and then look for those necessary attributes in candidates. I'd also rather have organisations that can train up staff internally and manage them properly. In that circumstance, an organisation could take more risks in who it hires to start with. But that seems like some kind of fantasy world.

TheFORCE

I'm surprised the author of the article hasnt posted the same for the ISACA certs.

ITHokie

OctalDump wrote: »

CISSP isn't meant to be...

It doesn't matter what we think it's meant to be - what's relevant is how it is perceived. If perception wasn't wildly out of whack, the controversy wouldn't exist. How often do you see discussions like this about Security+?

beads

Historically, the CISSP completed its objectives by providing a broad based, generalized overview of the entire security realm. The year the first CBK was, 1990 and the first sitting of the test was in 1994. Problem is the test nor the organization really kept up with the rapid pace of progress. Instead the ISC(2) choose to increase the number of certified "professionals" to meet industry demand not necessarily provide quality analysts. Audit appear from my experience to be far and few between, never heard of anyone IRL tell me they have been audited or rarely rejected though it has been occasionally reported through other channels. Never in meeting minutes. One of the tenants of a professional organization is self policing. We look pretty shaky on that point.

Yes, go dig around some, you'll find **** sites, brain ****, etc for any popular test - even for the beloved by HR, CISSP.

Third renewal in January! Wooohoo!

That 12 years for those who are counting.

- b/eads

goatama

OctalDump wrote: »

Personally, I'd much rather have organisations get to grips with what the actual skills/knowledge a job requires, and then look for those necessary attributes in candidates. I'd also rather have organisations that can train up staff internally and manage them properly. In that circumstance, an organisation could take more risks in who it hires to start with. But that seems like some kind of fantasy world.

Agreed. We recently put out for a security engineer, we said they needed 3 yrs with a SIEM, good networking knowledge, 5 yrs security experience, etc. Got tons of resumes with lots of CISSPs on them. So we did a phone screening first, asking basic knowledge questions (what tool would you use to verify data being sent is encrypted in transit, stuff like that). For the people that could pass the phone screening we brought them in for a technical exam. No multiple choice, all short answer. Told them we don't necessarily need the right answer, we just want to know how they think. Then the folks that passed that got a real interview.

We had over 200 applicants, ~30% had a CISSP. The first few rounds we had the applicants with the best resumes just come in for the test, no phone screening. None of them could get a single question on the written exam. So we revamped the process and added the phone screenings to see if we could weed out the "augmented" resumes. So with the next batch we had 15 phone screenings. Of those we had 2 people pass the basic phone screening and had them come in for the exam. Both did relatively well. Only one of them had a CISSP. We ended up hiring the guy without because he was more hands-on technical.

Danielm7

goatama wrote: »

None of them could get a single question on the written exam.

Ouch! I'm curious if you want to share a few of the written questions that no one could answer?

goatama

Danielm7 wrote: »

Ouch! I'm curious if you want to share a few of the written questions that no one could answer?

Well that would be telling, wouldn't it?

Honestly it's really basic stuff. Write a powershell script that loops through something. Small snippet of code to identify the security weakness. Here's this code, where would you put a SQi statement? Write a basic SQL injection. Here are some SIEM log entries, which entry is the most dangerous and why. Here's a pcap, where would you start. Stuff like that.

Again, the point isn't that they get the right answer (obviously, since some of it is open to interpretation), the point is to see how they work the problem. And we tell them that before they start. I was the last person we hired that got every answer. Most people don't even finish the test.

gespenstern

goatama wrote: »

Write a powershell script that loops through something. Small snippet of code to identify the security weakness. Here's this code, where would you put a SQi statement? Write a basic SQL injection. Here are some SIEM log entries, which entry is the most dangerous and why. Here's a pcap, where would you start.

Besides SIEM question you are doing it wrong. CISSP is a knowledge-based exam, it doesn't check skills. And it makes sense because essentially there are no experts in IT and everybody is a novice because everything is changing quickly and therefore you don't have time to become an expert. And before someone objects, an expert is someone who spent ~10 000 hours doing the same repetitive type of activity, such as playing chess or a violin or cropping corn.

Proper questions would be "how a cycle works in any programming language" or "how an SQL injection works" or "what a typical buffer overflow problem is", SIEM is legit (provided that vendor-specific messages are explained to the interviewee) and "how a typical network handshake works and overall IP stack works".

As Kelly Handerhan rightfully notes in her videos, a CISSP holder is a risk adviser. Who doesn't do much by themselves.

And it is more or less so in a real world, security folks rarely DO something compared to an infrastructure or development or network team, who do stuff all the day long. Security folks watch, analyze and advise.

Your questions are more for doers, who do repetitive tasks and therefore should have appropriate skills, such as writing a PowerShell code etc. There are other exams out there to test specific skills, such as OSCP (e.g. for SQL injection skills). Skills-based examinations have their shortcomings as well, BTW.

Therefore if a CISSP holder can't answer those questions it doesn't say much about quality of their CBK knowledge. You question more a knowledge or skills that are additional to the CISSP CBK. For example, asking to write a PoSh piece of code is a good question for an MCSA 2012 certified person. Asking to write a SQL injection is a good question for an OSCP certified person, etc.

goatama

I'm not sure if you read my original responses before you went about telling me I was "doing it wrong", but maybe you should before responding?

At no time did I say we were hiring "CISSPs". I said we were hiring for a security engineer. Someone with hands on experience. I didn't list here all of the requirements in the job posting, more a general idea. My point was also that someone with five years of *actual* security experience in two or more of the domains should be able to do all of those things. Or at least provide an educated response. Again, they didn't have to get all of the questions right, but more often than not they couldn't get ANY of them right. None. I also didn't list all of the questions (since, you know, Idk if any of you guys are ever going to apply for the next one that opens up).

So no, I'm not "doing it wrong", my point is that the CISSP is not the end-all-be-all cert that a lot of people seem to think it is.

billDFW

Isn't the CISSP geared more towards managers anyway, and less in-the-weeds ? The Secretary of the DOD is a manager, he does not need to know how to drive a tank or land on an aircraft carrier. He does however have people under him, who are Subject Matter Experts (SMEs) and have specific knowledge and talent on their specific tools.

He does however, need to understand those assets and how to deploy them, and use them.

My two cents...

TheFORCE

goatama wrote: »

I'm not sure if you read my original responses before you went about telling me I was "doing it wrong", but maybe you should before responding?

At no time did I say we were hiring "CISSPs". I said we were hiring for a security engineer. Someone with hands on experience. I didn't list here all of the requirements in the job posting, more a general idea. My point was also that someone with five years of *actual* security experience in two or more of the domains should be able to do all of those things. Or at least provide an educated response. Again, they didn't have to get all of the questions right, but more often than not they couldn't get ANY of them right. None. I also didn't list all of the questions (since, you know, Idk if any of you guys are ever going to apply for the next one that opens up).

So no, I'm not "doing it wrong", my point is that the CISSP is not the end-all-be-all cert that a lot of people seem to think it is.

I'd agree with gespenstern. You say you are not hiring CISSP's but you used the cert as probably something nice to have, preferred or required, in addition you compared the person with CISSP vs the one without and at the end you hired the one without because of the technical knowledge. Then again you compare the many CISSP and none of them seemed to know the answers, based on that you are assuming that the CISSP is not deemed worthy or it does not provide any real technical knowledge, the fact is CISSP does not provide technical knowledge and you should write your job descriptions aimed towards more technical people. Everyone knows that the CISSP is advertised as "an inch deep and a mile wide" certificate, you cannot expect them to know everything technical. In my opinion the strong skill a CISSP holder has is that they should know how to deduce and use logic to come up with a basic solution or basic analysis.

As an example, myself I had to create about 1000 AD security groups, started doing them 1 by 1, takes too long right? I understood what I had to do and how the process works. Didn't know any Powershell but I understood code, looked online at Microsoft and found a script, I used the script but it was not working, the script was an old version of PS and aimed to create groups for PowerPoint instead of AD. Once I realized that, I fired up the Powershell Help and found the parameters for the cmdlet NEW-ADGroup. Modified my script accordingly and the script worked on first try. It took me maybe 2 hours to identify and resolve the issue, it could have taken an MCSA maybe 5 minutes. My point is and what gspenstern was trying to say is that CISSP are problem solvers using logic, not necessarily implementations or specific technologies, even though some very skilled once can be both.