
You've got a client release next week, an auditor asking for evidence, and three vendors pitching three different tests. One promises a black box assessment, another sells continuous scanning, a third talks about autonomous agents. They all sound thorough, and they all cost money. The hard part isn't buying a test. It's knowing what each model can prove, what it'll miss, and how you'll check its findings before you hand them to a client. That's the lens we'll use for every model below.
Table of Contents
- Models of Testing at a Glance
- Black, Grey and White Box Testing by Tester Knowledge
- Manual, Automated and Hybrid Testing Compared
- Continuous Versus Point-in-Time Testing Cadence
- Agentic Testing and How to Verify Its Findings
- Confirm Scope and Authorization Before Any Test Starts
- Choosing a Testing Model for Each Situation
Models of Testing at a Glance
Start with the seven common models of testing side by side. Read across each row for what the tester knows, where the model fits, and where it breaks down.
Penetration testing models at a glance
| Model | Tester knowledge | Typical use | Main strength | Main limit |
|---|---|---|---|---|
| Black box | Target name only | External perimeter, outsider view | Realistic attacker view | Slow to reach deep logic |
| Grey box | Partial access such as test accounts and API docs | Authenticated web and API tests | Reaches roles and workflows quickly | Limited to the roles and docs provided |
| White box | Full access to code, configs and architecture | Code-level and configuration review | Finds root cause | Costly and needs senior skill |
| Manual | Human judgement at every step | Business logic and chained attacks | Creative, context-aware reasoning | Hard to repeat and scale |
| Automated | Scripted tools and rules | Known vulnerabilities across many assets | Fast, repeatable coverage | Misses novel attack paths |
| Continuous | Recurring tests on a schedule | Changing environments and frequent releases | Early detection of drift | Pays off only with a remediation workflow |
| Agentic or autonomous | AI agent plans and runs tests | Large estates and repeated retesting | Broad coverage in hours | Needs evidence checks and human review |
These aren't rival products. They're overlapping lenses, and one engagement often combines several. You might run a black box external test to see the perimeter as a stranger sees it, then a grey box authenticated review to test what logged-in users can do, then automated retesting to confirm fixes a week later.
Think of checking a house. A stranger can walk the outside and try the doors. An inspector with the floor plan checks rooms in a sensible order. A structural engineer with full drawings can explain why a wall is cracking. A sensor reports every week whether something changed. Each check answers a different question, and none replaces the others.
Three Questions That Narrow the Choice
- What access does the tester have? No credentials, test accounts, or source code changes what can be reached in the time you have.
- How often does the target change? A stable system suits a scheduled deep test. A system that ships weekly needs recurring checks between deep tests.
- Who checks the results? A finding you can't reproduce, evidence, and retest isn't a finding yet. It's a claim.
Note the two or three rows that match your situation now. You'll use them as a filter while each model gets explained in detail, with its verification burden made explicit.
Black, Grey and White Box Testing by Tester Knowledge
Picture three burglars. The first has no map and only knows the address. The second has a floor plan. The third has the keys, the alarm code, and the building drawings. What each one finds, and how fast, depends on what they started with. Tester knowledge works the same way.

Black Box Starting From Outside
A black box tester starts with a target name and works from outside, mirroring an unauthenticated attacker. The early work is quiet. The OWASP Web Security Testing Guide separates passive testing, where the tester explores as an ordinary user and observes requests and responses through an HTTP proxy, from active testing across 12 named categories including authentication, authorization, session management, input validation, business logic, and API testing.
That passive phase builds your inventory first. Endpoints, parameters, cookies, headers, and API calls get mapped before anyone sends an intrusive payload. Skip it and you test the login page hard while missing the undocumented API route the mobile app uses.
Grey Box With Accounts and Documentation
Grey box testing adds test accounts, API documentation, or a role matrix. That small head start changes what you can prove. Say you log in as two ordinary users. You change an object ID in a request and user A can suddenly read user B's invoice. That's broken access control, shown with two accounts and one edited request.
A black box tester might spend hours creating accounts, guessing the workflow, and finding that same endpoint. The grey box tester goes straight to the question that matters. Does the application check ownership on every request, or does it trust the ID the browser sends?
White Box With Code and Configuration
White box testing gives the tester source code, architecture diagrams, and configuration files. Now the goal shifts from demonstrating an attack to finding its root cause. The tester can spot the missing authorization check in the code path, trace every route that shares it, and flag a hard-coded secret that no external test would ever reveal.
More knowledge finds deeper flaws faster, but it costs senior time and can blind you to what an outsider actually sees.
That's the honest trade-off. White box work is slower to staff and more expensive per target. It also answers a question black box testing can't. Is this one broken endpoint, or is the same missing check copied across forty endpoints?
Don't treat a thin black box report as proof of security. It proves only that an outsider with that time and that access found little. Every report should state the access level used, the accounts provided, and what was out of scope, so a reader can judge what the result means.
Manual, Automated and Hybrid Testing Compared
Say a tester applies a discount code, completes checkout, then edits the cart and checks out again. The code applies twice. No scanner flags it because every individual request looks valid. The flaw lives in the sequence, and a person who understands how the shop should behave spots it. That example explains the whole manual versus automated argument better than any feature list.
Manual testing is a person reading responses, forming a hypothesis, and chaining small weaknesses into one real attack. A low-severity information leak plus a predictable reset token plus a weak session check becomes account takeover. Each step alone looks minor.
Automated testing runs scripted checks and known-vulnerability tools across thousands of endpoints in hours, then repeats identically next week. Automation is already normal practice. The ISTQB Worldwide Software Testing Practices report gathered more than 3,200 responses from professionals in 89 countries in 2015 to 2016 and described test automation as a major industry priority.
Adoption numbers need care, though. A literature review of industrial surveys on software testing practices reports test-automation adoption at approximately 72%, while in about 40% of surveyed cases the proportion of automated test cases exceeded 20%.
Automation Is Not One Model
The same review breaks automation down by activity, and the spread is the point. Rates were 36.4% for test execution, 32.5% for test management, 33.8% for unit testing, 15.6% for static analysis, and only 13.0% for test design. Running tests is easy to automate. Deciding what to test, interpreting risk, and exploring the unknown aren't.
Model-based testing sits between the two camps. You describe expected behaviour as states, transitions, and constraints, then generate sequences that walk those transitions for API and workflow tests. Research on dynamic model-based testing found purely random traversal ineffective for continuous-integration-scale testing, while a greedy strategy achieved an error-detection rate comparable to precalculated optimal paths on large models. In plain terms, prioritise risky transitions instead of sampling the state space evenly.
Manual versus automated testing by criterion
| Criterion | Manual testing | Automated testing |
|---|---|---|
| Business logic flaws | Strong, uses human reasoning | Weak without custom rules |
| Speed on large estates | Days to weeks | Hours |
| Repeatability | Changes with tester and note quality | Identical on each run |
| Cost per target | High, senior time | Lower per run after setup |
| False positives | Lower with a skilled tester | Higher until verified |
| Novel attack chains | Strong | Weak |
If you want the practical delivery trade-offs explored further, our guide to manual vs automated penetration testing works through cost and staffing in more detail.
The Hybrid Pattern Most Teams Land On
Hybrid testing is the usual answer because each side covers the other's blind spot. Automation handles discovery, known-vulnerability checks, and regression. Humans reproduce each serious finding, test the workflows scanners don't understand, and decide whether a result matters in context.
Practical rule: Never report an automated finding that a tester hasn't reproduced.
Reproduction means running the request again, confirming the impact with your own eyes, and capturing the evidence. A scanner banner saying a service looks vulnerable isn't proof. The request, the response, and the resulting access are proof.
If you're starting from zero, automate three things first. Asset discovery, so you stop missing hosts. Known-vulnerability checks, so common flaws surface fast. Retesting after fixes, so closure is verified rather than assumed. Keep human time for authenticated workflows, payment logic, and anything where the word "should" appears in the requirement.
Continuous Versus Point-in-Time Testing Cadence
Is more frequent testing automatically better? That's the assumption worth questioning. Frequency only reduces risk when someone fixes what the tests find, and plenty of programmes fail exactly there.
A point-in-time test is a snapshot. You schedule it before launch or for an annual assessment, and you get a clear baseline plus an audit-ready report. Its weakness is staleness. The report describes the system as it was, and the next release changes the answer.
Continuous testing runs recurring automated checks after each deployment or on a fixed cycle. It catches drift between engagements, such as a new exposed service or a reverted configuration. Our overview of continuous penetration testing covers the operating model behind that cadence.
What the Remediation Numbers Show
A 2026 analysis of continuous penetration testing outcomes covering more than 16,500 tests across nearly 3,000 organizations found the top tenth reduced half of high-risk findings in about 10 days, while the slowest tenth took 249 days. Organizations using programmatic testing were 4.5 times more likely to resolve critical findings within three days than ad hoc or compliance-driven teams.
Read those numbers carefully. The gap between 10 days and 249 days isn't a scanning gap. It's an ownership, prioritisation, and retesting gap. The fast group didn't just test more. It closed the loop.
Say weekly scans feed a ticket queue nobody owns. Every Monday the same critical finding appears, gets a new ticket number, and ages. Your test count looks impressive. Your risk hasn't moved. An auditor who asks how long that finding stayed open will see the difference immediately.
Measure Closure Instead of Test Count
- Time to verified fix: Days from confirmed finding to a passed retest, not to a developer saying it's done.
- Critical-finding closure rate: The share of criticals closed inside your agreed window.
- Retest pass rate: How often a claimed fix survives an actual retest on the first attempt.
- Named owners: A person, not a team alias, accountable for each open critical.
Choose the cadence your remediation team can close, then increase it. A monthly cycle with verified fixes beats a weekly cycle that creates backlog. Increase frequency only when closure keeps pace.
Agentic Testing and How to Verify Its Findings
An agentic test follows a sequence you can audit step by step. Scope is set first. The agent then runs reconnaissance, plans its attacks, executes them with tools such as Nmap, SQLMap and Nuclei, verifies what it finds, and writes the report. Our explainer on agentic AI in cybersecurity describes how that orchestration works under the hood.

The strengths are concrete. Agents run checks in parallel, cover every endpoint and parameter consistently instead of sampling, and return results in hours instead of weeks. For a service provider retesting many clients after patch cycles, that throughput is the whole business case.
The limits are just as concrete. Business logic, creative attack chains, and organisational context are weaker. An agent can confirm that a reset token works twice. Judging whether that's fraud-enabling in your client's checkout flow still needs a person. And no model changes one basic fact. Security testing cannot identify every vulnerability, and any service is provided "as is" and "as available" on that understanding.
The Test Oracle Problem
Every test needs an oracle, the mechanism that decides whether a result is correct. Research on the test oracle problem in autonomous systems identifies the oracle as the central unresolved issue when systems are nondeterministic or autonomous. For agentic testing the question is blunt. Who decides that this result is correct, safe, and sufficiently evidenced?
Compare oracle types rather than trusting one. Deterministic assertions check exact expected behaviour. Runtime evidence captures what happened. Exploit confirmation proves access rather than inferring it. Human review judges context. Retesting after remediation proves the fix. A strong programme layers several of these instead of accepting an agent's summary sentence.
An anomalous response is not a verified exploit. Treat it as a lead until the evidence chain is complete.
The Evidence Chain to Demand
- Trigger: The exact request, payload, or sequence the agent sent, with the account and role used.
- Response: The server reply showing the behaviour, not a paraphrase of it.
- Impact: What the attacker gains, such as another user's record, a session, or a configuration change.
- Remediation context: Where the flaw sits and what a fix must change, so a developer can act without guessing.
You can also measure an agent instead of trusting its marketing. Seed known flaws into a test application, such as a removed authorization check or an altered boundary condition, then see whether the agent catches them with evidence. That's the mutation-testing idea applied to security agents. A high endpoint count means nothing if planted flaws survive.
Reject any agent finding that lacks a complete evidence chain. Send it back for reproduction, downgrade it to an unverified lead, or drop it. Better autonomous testing depends less on generating more test cases than on building auditable judgement around each result.
Confirm Scope and Authorization Before Any Test Starts
No model starts until scope is settled. NIST Special Publication 800-115 defines penetration testing as four phases, planning, discovery, attack, and reporting, and requires a written Rules of Engagement before testing begins. That document specifies what's in scope and what's permitted, and it authorizes the defined activities so testers don't seek fresh permission for each action.
Follow these steps in order.
- List every target with its owner and hosting provider, including cloud tenants and SaaS products.
- Write the ROE with allowed techniques, forbidden actions, and testing time windows.
- Collect proof of authority such as an ownership record, a signed authorization, or domain control validation.
- Separate discovery from attack so you map the target and analyse weaknesses before any exploitation.
- Set mode and schedule before launch, including black box, grey box, or white box access.
- Agree stopping conditions and name the person notified if a service fails during testing.
- Run one small test and review its evidence before the full engagement proceeds.
Third-party clouds, hosts, and SaaS products need their own written testing permission under those providers' policies. On our platform, tests run only against targets the customer defines and authorizes, and for each engagement the customer must confirm and attest that they're authorized to test the specified targets and scope. If you want to see the workflow first, you can create a free account at portal.threatexploit.ai/register, explore the platform, and run a test pentest to see results firsthand. The written ROE is what your model choice must fit inside.
Choosing a Testing Model for Each Situation
Match the situation first, then argue about tools. This table gives the starting recommendation most experienced teams would reach for.
Recommended testing model by situation
| Situation | Recommended model | Why |
|---|---|---|
| Annual PCI DSS 11.3 assessment | White or grey box with manual exploitation and a retest | Requires external and internal testing plus repeat tests until access is blocked |
| Pre-release web app or API check | Grey box with automated checks in the pipeline | Authenticated coverage on every build |
| Large external attack surface | Black box automated discovery with manual triage | Broad discovery, while triage keeps noise down |
| Complex business workflows | Manual or grey box testing | Logic flaws need human judgement |
| Ongoing exposure monitoring | Continuous automated testing with scheduled manual deep dives | Early detection plus periodic human depth |
| Many clients, small team | Agentic testing with human sign-off | Throughput across clients with review of critical findings |

Four Criteria That Decide It
Compliance obligations come first because they can dictate method, not just outcome. Attack surface size decides whether human-only coverage is even arithmetically possible. Release frequency decides how fast a snapshot goes stale. Available senior capacity decides how much manual depth you can honestly deliver instead of merely promise.
Take the compliance case. PCI DSS penetration testing guidance for Requirement 11.3 distinguishes a vulnerability assessment, which identifies and reports weaknesses, from a penetration test, which attempts to exploit them to determine whether unauthorized access is possible. Testing must cover the cardholder data environment and connected systems, include external and internal testing, and repeat after correction until the access is no longer possible. Automated scanning alone can't satisfy that because it lists weaknesses without proving or disproving access.
Contrast a small web team shipping every week. It doesn't need an annual white box marathon. It needs grey box testing of authenticated APIs plus automated checks in the build pipeline, so role and object-level flaws surface on the build that introduced them. A quarterly manual session then hunts the workflow logic the pipeline can't judge.
A service provider scaling across many clients faces a different constraint, senior hours. Agentic testing supplies breadth across every client's estate, and human sign-off on every critical finding supplies the judgement clients and auditors trust. The provider sells verified results, not raw agent output.
For most organisations the best answer is a hybrid mix, chosen deliberately rather than by vendor default. Pick the model that covers your riskiest asset at an access level you can formally authorize, verify its findings with an evidence chain, and retest until closure is proven. A smaller test you can defend beats a larger one you can't.
If you're weighing these models for client work, ThreatExploit AI runs autonomous penetration tests across web, network, and cloud targets with evidence-backed, compliance-mapped reports your team can review and retest against. You define and attest the scope, the platform handles reconnaissance through verified reporting, and your seniors keep sign-off on what reaches the client. Visit ThreatExploit AI to see how that fits your delivery model.