Pen-testing vs CTFing

edited December 2017 in Website

So, I spent a lot of my spare time here since October (the month, not the machine :P) and I have a question: is this site more about pen-testing or about CTFing?

Of course I learned a lot of new things and techniques, I had the chance to practice other things that I learned a while ago and, for sure, I also enjoined my time: some machines I've done are really cool (Calamity and Jail really rock).

But now I'm starting to realize that most of the online machine are - at least to me - really far from what I see every day in the real world. Let me say too "CTFgish". And I wonder if the time spent on them worth the knowledge learned. Let me do an examples: what I've really learned from dirbusting with that bigger wordlist that "easy" machine? Or do I really ever find creds hidden in that steg-something file on a production server?

Machines - I read in other posts the forum - are getting harder. For sure this let gurus and omniscients to fight for being the best. But how are getting harder? And are they fighting to be the best pen-tester or the best CTFger?

May be that a realworld/ctf value or ratio (evaluated when submitting a flag) would be useful to users (and site owners too) to understand what we all are doing here.

0xEA31

Comments

  • edited December 2017

    As an OSCP holder and red team / penetration tester of several years, I feel HTB is the equivalent to me as a Sunday papers Crossword puzzle is to an English teacher. Ye, It's awesome to learn new tricks and techniques as someone in the field, but none of this is going to help me on the daily really.

    The harder machines are all stego / crypto based exercises in which knowledge/experience in CTF style challenges trumps logic, talent and job relatable tasks (looking at you Shrek)

    No job interview is going to question you on your ability to invert two audio tracks to find the SSH key to steal the crown Jewels.

    But they will ask you about Eternal Blue!

    Having said that, I will push HTB as a learning and development path for my team for 2018 - but I'll be selective in the boxes to target as the time/value ratio is waaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaay off on some of these machines.

  • I really like how silence put it, it's the Sunday papers crossword puzzle. To your point CTF vs Real-World, I'd love to have nothing but real-world boxes, however it is hard to consistently put out boxes like that. They take more time, and with the scope being much smaller take longer to think up. If we just took the approach as some platforms and just migrated the latest CVE into the lab, it would be waaaaaay to easy.

    I am also against tagging boxes with what is involved, as that is a huge part of the recon phase. There's a misconception many people have about "red teaming". In my opinion, it's more about misconfigurations than actual exploits. If I see Eternal Blue/DirtyC0w/etc on an engagement, I just take note and keep looking. Not going to use any exploit when I can get in through a misconfiguration.

    The "CTF" problems show the ability to critically think, which is useful. Some machines, I dislike and think could have been done a bit better to minimize the pain that comes with it being a "CTF Problem". But I wouldn't say any aren't worth doing. The two audio file problem is how "Noise-Cancellation" works in general and does have few practical uses. Also, it's just a really cool thing to know.

    I would say machines are getting harder because of people really pushing the limits of what can be done from a "Boot2Root" machine. There was a wave of lateral movement machines using things like Qemu/docker/etc. The bar is constantly being raised for what a good machine consists of. I completely disagree with the harder machines being stego/crypto based, that use to be the case but I don't think it is anymore.

  • @ippsec said:
    I am also against tagging boxes with what is involved, as that is a huge part of the recon phase.

    I'd not tag the machines with what is involved. That's why I propose is an user-driven overall evaluation when submitting a flag. May be that, let's suppose, the recon phase for a machine is pure CTF style and the priv-esc is a real-world one.

    @ippsec said:
    The "CTF" problems show the ability to critically think, which is useful. Some machines, I dislike and think could have been done a bit better to minimize the pain that comes with it being a "CTF Problem".

    You are right and, to me, that's the tip of the iceberg. If these machines could have been better, why they weren't?

    Here you are a quote from a blog (link here if you are interested):

    from a blog:
    [...] Security researchers entering the industry post-2010 learn almost exclusively via Capture the Flags competitions [...].

    That's scaring. Isn't gamification gone a little bit too far away? Should we measure, somehow, how far it has gone?

    Please note, I'm not a pen-tester. But since sometimes I ask for pen-tests and sometimes even review pen-tests, I must understand the real value of the bunch of pages they deliver to our CEO, CIO or whatever. HTB seems to be the perfect place to see how the pen-testing community evolved and evolves.

    Call me "old fart" if you want, but what we need are skilled people with a pen-test mindset and competences. Sure they should be open-minded, critically-thinking ones.

    But we don't need CTFgers that call themselves pen-testers.

    It's not the same.

    0xEA31

  • @0xEA31 said:

    @ippsec said:
    The "CTF" problems show the ability to critically think, which is useful. Some machines, I dislike and think could have been done a bit better to minimize the pain that comes with it being a "CTF Problem".

    You are right and, to me, that's the tip of the iceberg. If these machines could have been better, why they weren't?

    Plenty of reasons but really it comes down to:

    • Machines are user submitted and everyone has different opinions on what they like. Just because I think it could have been better, doesn't mean my solution should be implemented.
    • This isn't anyone's fulltime job and even if it was -- People aren't perfect hindsight, is 20/20.
    • One machine a week is really tough. It was proposed to go to every other week a few months ago and people were against that. Since then everything about the site has grown and hopefully, that allows for better machines in 2018.

    0xEA31 said:
    Here you are a quote from a blog (link here if you are interested):

    from a blog:
    [...] Security researchers entering the industry post-2010 learn almost exclusively via Capture the Flags competitions [...].

    That's scaring. Isn't gamification gone a little bit too far away? Should we measure, somehow, how far it has gone?

    I don't really think its gone too far. I don't even have an associates degree, CTF's and passion projects are pretty much the only way I've learned. That being said I wouldn't ever use a person's CTF History as an indicator of skill. It would be used for a willingness to learn outside of work hours which is probably more important.

    0xEA31 said:
    Please note, I'm not a pen-tester. But since sometimes I ask for pen-tests and sometimes even review pen-tests, I must understand the real value of the bunch of pages they deliver to our CEO, CIO or whatever. HTB seems to be the perfect place to see how the pen-testing community evolved and evolves.

    It's impossible to say anything without knowing the people that created the Pentest you dislike. I think a bigger problem is the "puppy mill" pentest shops that just run automated tools and call it a day. That isn't a problem created by people getting experience from CTF's, I'd put money on the people doing that don't even do CTF's.

  • Just my 2cents (im not a pentester or professional),
    if you design a box and you don't have any other context like a real company has with employees and stuff which gives you multiple angles to get creds and usernames, you need an entrypoint to this machine.
    So if you don't want to use yet another $webshellable-service as an entry point, you have to hide it in a not so obvious way, if you don't want it to be metasploitable.

    Also a lot of privesc's are awesome in a sense of how a service or cronjob could be missconfigured and later missused.
    IMHO using a poorly written bash script from a lazy/overworked admin to clean logfiles or else, is more real world than using a kernelexploit to get root. Also i love the new trend to containerize applications.

    Only thing i really disklike is if you use a specific wordlist for a bruteforce task. The challenge does not get any cooler if you use a wordlist nobody knows about, they only get frustrated.

  • I see this website more of a CaptureTheFlag competition.

    I think a lot of it comes from the mindset and thinking critically. I don't pretend to be the best and amazing "hacker" out there, but the challenge is what keeps me going and when I figure it out, I feel like the best and most amazing hacker out there and I think that's important. There are times where I really just want to know the answer and admit defeat, but that feeling when you figure it out despite that, it is Oh. So. Good.

    This is not nearly as technical as pen-testing, that requires reports, approval, POCs, evidence, so much stuff that most of your time is spent not doing the thing that you got into the field for. So at the end of the day, I want to come home, bang my head against a brick wall, crush a machine or review things I did in past challenges, without all the "bullshit".

    Ippsec said something above which I really agree with, the part about taking the time outside of work/school to learn and flex that all important muscle.

    There are some valid techniques though that can be learned from CTFs and applied to Pen-Testing and I think the vice versa is true as well. I think they go hand in hand, but not one in the same.

  • @s1lence said:

    No job interview is going to question you on your ability to invert two audio tracks to find the SSH key to steal the crown Jewels.

    very late reply but, as the maker of this challenge, I have to say this was absolutely on point :tongue:
    there are many fully realistic machines however !
    stego and CTF-ish SQL injections appear of very few of them :)

  • Maybe the rastalabs are more like pro pentesting than the normal boxes??

    ompamo

  • edited June 2018

    I was actually going to link this article. It really sums up a lot of thoughts I've gathered over the past years.

    I've been doing Penetration Testing for a couple of years professionaly. Most tests have been performed in highly secured environments - mainly banking, finance, exchanges, medical. It's very rare when I'd have to come up with a complex hack akin to what we have on CTF challenges. From the technical standpoint, I think the benefit of CTF's is when you actually learn new things as you're trying to score points rather than the puzzle solving activity itself.

    What CTF's will fail to teach you, in my opinion, are the following:

    • ** Thinking about the process rather than just the goal**. A penetration test is much more than just taking over windows domains and popping shells. It's a professional assessment aimed at improving technology built by other professionals which are in turn used by businesses and organizations. Yes you can hack humans too, but even then, you're testing organizational/operational security - not your ability to have someone click on a cat image.

    • ** Penetration Testing is a service indented for a customer**. As a result, you should be able to understand what your own company does as well as the customers in order to deliver results with value. No one cares how cool the hack is (apart from fellow professionals). It's all about the value that your testing brings to your clients.

    • ** Discipline.** Just look at how many resets the community here does per box. In a real world scenario, especially if youre testing production systems, you can cause a lot of damage to your client, your company, and your personal reputation if you're not careful when testing. You can even incur financial loss all the way to jailtime.

    • ** Non-Technical Communication.** Writeups are fine and dandy but they do not prepare you for communicating your finings to all other stakeholders (Managers, Executives, Developers, etc.) Communication skills, from what I've seen, make you a very valuable asset in a real world scenario. A lot of the times things are not as simple as "you just need input validation". Very often you have lots of parties involved in the process, and everyone views your results differently - often implicating an entire organization to your process. Your communication can save or damage people and assets in the millions.

    I could go on but those are just a few of the top of my head. I joined HTB very recently, and im thoroughly enjoying these challenges, however they do not reflect or equip you with a proper skillset to hit the PT market.

    The community here seems awesome :)
    See you all on the charts!

  • +1 to @XXYXZX in a pentest you have to find all issues as possible and give advice in how to fix and have a 2 way conversation with the clients, on CTF you just need to get the flag and bye.

    Hack The Box

    • ** Non-Technical Communication.** Writeups are fine and dandy but they do not prepare you for communicating your finings to all other stakeholders (Managers, Executives, Developers, etc.) Communication skills, from what I've seen, make you a very valuable asset in a real world scenario.

    I do love read about real world experience.
    From my sys admin point of view, I guess that CTF and real world scenario are the best recipe to keep growing and become a focused pen-tester.

    It could improve the way we solve CTF here adding the "communication skill".
    Adding the REPORT as part of CTF will help anyone to get better and closer to the real world of pen testing. I am sure ippsec will love it.

    Marbew

  • @marbew said:

    • ** Non-Technical Communication.** Writeups are fine and dandy but they do not prepare you for communicating your finings to all other stakeholders (Managers, Executives, Developers, etc.) Communication skills, from what I've seen, make you a very valuable asset in a real world scenario.

    I do love read about real world experience.
    From my sys admin point of view, I guess that CTF and real world scenario are the best recipe to keep growing and become a focused pen-tester.

    It could improve the way we solve CTF here adding the "communication skill".
    Adding the REPORT as part of CTF will help anyone to get better and closer to the real world of pen testing. I am sure ippsec will love it.

    Well said!

  • +10 to @XXYXZX
    Totally agree

  • @XXYXZX
    Very well said...

  • I totally disagree with sunday cross words.

    peek

  • @XXYXZX said:

    I was actually going to link this article. It really sums up a lot of thoughts I've gathered over the past years.

    I've been doing Penetration Testing for a couple of years professionaly. Most tests have been performed in highly secured environments - mainly banking, finance, exchanges, medical. It's very rare when I'd have to come up with a complex hack akin to what we have on CTF challenges. From the technical standpoint, I think the benefit of CTF's is when you actually learn new things as you're trying to score points rather than the puzzle solving activity itself.

    What CTF's will fail to teach you, in my opinion, are the following:

    • ** Thinking about the process rather than just the goal**. A penetration test is much more than just taking over windows domains and popping shells. It's a professional assessment aimed at improving technology built by other professionals which are in turn used by businesses and organizations. Yes you can hack humans too, but even then, you're testing organizational/operational security - not your ability to have someone click on a cat image.

    • ** Penetration Testing is a service indented for a customer**. As a result, you should be able to understand what your own company does as well as the customers in order to deliver results with value. No one cares how cool the hack is (apart from fellow professionals). It's all about the value that your testing brings to your clients.

    • ** Discipline.** Just look at how many resets the community here does per box. In a real world scenario, especially if youre testing production systems, you can cause a lot of damage to your client, your company, and your personal reputation if you're not careful when testing. You can even incur financial loss all the way to jailtime.

    • ** Non-Technical Communication.** Writeups are fine and dandy but they do not prepare you for communicating your finings to all other stakeholders (Managers, Executives, Developers, etc.) Communication skills, from what I've seen, make you a very valuable asset in a real world scenario. A lot of the times things are not as simple as "you just need input validation". Very often you have lots of parties involved in the process, and everyone views your results differently - often implicating an entire organization to your process. Your communication can save or damage people and assets in the millions.

    I could go on but those are just a few of the top of my head. I joined HTB very recently, and im thoroughly enjoying these challenges, however they do not reflect or equip you with a proper skillset to hit the PT market.

    The community here seems awesome :)
    See you all on the charts!

    Very true points!

    A part 2 to a few bullets (what CTF's do not teach...and what many CTF's do not realize), is that the entire premise of pen testing is generally risk validation. The client organization is being required to perform assessing in order to identify, validate, and further understand risks and issues in their vulnerability management programs. The underlying process of vulnerability management is serving as a higher level business function and all the wizardry is to achieve that end-state of meeting the goals of that process.

    | OSCP (OS-40299) | WCNA | CCNP | CCDP | ECSAv9 | CEHv8 | CISSP | Sec+

  • Ctf/labs/certifications: hard, challenging,entertaining,learning new stuff, enthusiasm of a child
    Real world: boring, boring like an adult, boring like a business, rinse repeat with a lot of caveats all the time
    Job ruins everything it touches

    halfluke

  • Alot of awesome stuff here. The one thing I will add is that I have directly used some techniques I have learned here in my Pentest role. I feel I have learned a method of thinking about problems that I didn't have prior to HTB. Overall it is entertaining and a fun game.

  • Also as someone else said, the HTB community is the best one I'm in. People always ready to answer questions or give a little nudge. I pay for VIP as much to support the community than anything else.

  • It might be a point of debate for all of you (pentesters and CTF players).
    But for me the HTB platform is something that makes me feel valuable and also feel like being part of something bigger & better than myself.

    The people on this platform are elites in their own way of pentesting/CTFing about pwning the boxes. Each one has something to share/teach and i feel like the student who want's to grab all this things.

    You might call me hobbyist cause i neither have job in this field nor played CTF's except MSF that was only one i participated in.

    It's may be true that in real life scenarios no-one uses steg or crypto thing to hide creds but it may force you to level up the way you think of the problem either it forces you look into details and never miss anything on the way.

    I also hate when some boxes make you use specific wordlist to find things or some boxes requires a whole new level of out of box thinking. I failed miserably from time to time but that's what makes me keep going.

    It'll take time to meet every one expectation but if you need to speed up things it's good to provide feedback's and contribution.

    No one knows Everything but Everyone knows Something different. If we keep on contributing the things will definitely going to change/ level up.

    B0rN2R00T

  • Type your comment> @halfluke said:

    Ctf/labs/certifications: hard, challenging,entertaining,learning new stuff, enthusiasm of a child
    Real world: boring, boring like an adult, boring like a business, rinse repeat with a lot of caveats all the time
    Job ruins everything it touches

    let's put some punk fun in that :)

    peek

  • edited March 14

    Type your comment> @peek said:

    Type your comment> @halfluke said:

    Ctf/labs/certifications: hard, challenging,entertaining,learning new stuff, enthusiasm of a child
    Real world: boring, boring like an adult, boring like a business, rinse repeat with a lot of caveats all the time
    Job ruins everything it touches

    let's put some punk fun in that :)

    Well, I'm exaggerating because I'm facing the reality of pentesting for the first time.
    And the more I grow old, the more I would like to go back to childhood. But I'm a pretty hopeless case...
    Only, I cannot accept people who frown upon CTFs and other learning platforms, because they are not "real life". Don't they see that reality sucks? :)

    halfluke

  • Type your comment> @halfluke said:

    Type your comment> @peek said:

    Type your comment> @halfluke said:

    Ctf/labs/certifications: hard, challenging,entertaining,learning new stuff, enthusiasm of a child
    Real world: boring, boring like an adult, boring like a business, rinse repeat with a lot of caveats all the time
    Job ruins everything it touches

    let's put some punk fun in that :)

    Well, I'm exaggerating because I'm facing the reality of pentesting for the first time.
    And the more I grow old, the more I would like to go back to childhood. But I'm a pretty hopeless case...
    Only, I cannot accept people who frown upon CTFs and other learning platforms, because they are not "real life". Don't they see that reality sucks? :)

    keep that spirit, life is so short

    peek

Sign In to comment.