How to Choose Samples for Control Testing Without Weakening Assurance

Sampling can make control testing practical.

It can also make assurance weaker if it is done poorly.

A tester selects a few changes.
A reviewer checks a handful of access approvals.
A SOX team samples reconciliations.
A SOC 2 team samples user access reviews.
Internal audit samples vendors.
Compliance samples policy attestations.
Cyber risk samples vulnerability remediation.
A GRC team samples evidence to avoid testing every record.

That can be appropriate.

But sampling is not the same as guessing.

Sampling is not picking the easiest records.
It is not picking whatever evidence is available.
It is not testing three examples because three feels manageable.
It is not avoiding difficult populations.
It is not using last quarter’s sample without thinking.
It is not treating a small sample as proof of the whole control environment.

Sampling should be designed.

It should be tied to the control objective, population, risk, framework, evidence, test procedure, and conclusion.

A good sampling approach helps teams answer:

• What population are we testing?

• Why are we sampling instead of testing everything?

• Which items are higher risk?

• How many items should we test?

• How were the items selected?

• What attributes are we testing?

• What happens if we find exceptions?

• Does the sample support the conclusion we want to make?

• Does the sample work for SOX, SOC 2, ISO, NIST, internal audit, or compliance testing?

• What evidence proves the sample was selected and tested properly?

A weak sample creates false confidence.

A strong sample creates efficient assurance.

Connected GRC helps by linking sample design to controls, populations, evidence, test results, issues, remediation, validation, dashboards, and decisions.

The goal is not to test less.

The goal is to test intelligently.

What is sampling in control testing?

Sampling in control testing is the process of selecting less than 100% of a control population to evaluate whether a control operated as designed during a period, for a scope, and against defined test attributes.

A sample may be used to test:

• user access reviews

• privileged access reviews

• change approvals

• vendor reviews

• incident response records

• vulnerability remediation records

• reconciliations

• management review controls

• policy attestations

• privacy assessments

• AI use-case approvals

• business continuity tests

• control owner evidence submissions

• exception handling

• remediation validation

PCAOB AS 2315 defines audit sampling as applying an audit procedure to less than 100% of items in a population for the purpose of evaluating a characteristic of that population. It also notes that both statistical and nonstatistical approaches require professional judgment and can provide sufficient evidence when applied properly.

That principle applies beyond formal audit sampling.

Even in compliance testing, internal monitoring, control self-assessments, or GRC program testing, sampling should be deliberate, documented, and tied to the conclusion being made.

Why sampling decisions matter

Sampling decisions matter because a sample can shape the conclusion.

If the sample is too small, biased, poorly documented, or disconnected from the real population, the test result may not be reliable.

Poor sampling creates problems such as:

• testing the wrong population

• missing high-risk items

• drawing broad conclusions from narrow evidence

• failing to identify repeat exceptions

• weakening SOX or SOC 2 evidence

• under-reporting control failure

• creating disputes with auditors

• misleading executives

• creating dashboards that look green but are unsupported

Strong sampling helps teams:

• focus testing on risk

• reduce unnecessary control-owner burden

• preserve audit support

• identify exceptions earlier

• understand root cause

• support retesting and validation

• reuse evidence responsibly

• report control health with confidence

Sampling is a judgment call.

But it should not be an undocumented judgment call.

Sampling is not always appropriate

Before sampling, ask whether sampling is appropriate at all.

Sometimes the right answer is to test 100% of the population.

Sampling may not be appropriate when:

• the population is small

• the control is high-risk

• the control is automated and full-population testing is feasible

• prior failures are common

• the control supports a key SOX assertion

• the control involves critical vendors

• the population contains few but high-impact items

• every exception is material

• the control is new or recently changed

• evidence quality is weak

• the organization needs full coverage for management, audit, or regulatory reasons

Example:

If there are eight high-risk vendors onboarded during the year, sampling two may not be enough. Testing all eight may be more appropriate.

If there are five critical changes to a financial reporting system, testing all five may be more defensible than sampling one.

If there are thousands of low-risk policy attestations, sampling may be reasonable.

The first question is not:

How many should we sample?

The first question is:

Should we sample at all?

The Control Testing Sampling Model

A practical sampling model has ten steps:

This is not only an audit exercise.

It is a Connected GRC workflow.

Step 1: Define the control objective

Sampling starts with the control objective.

You cannot choose a sample properly if you do not know what the control is supposed to prove.

Example control objective:

Only authorized users have access to production systems.

Possible sample population:

• all users reviewed during the quarter

• all access reviews completed during the period

• all applications in scope

• all privileged users

• all access changes during the period

Those are different populations.

The right one depends on the control objective and the test procedure.

Another example:

Control objective:

High-risk vendors are reviewed before onboarding or renewal.

Possible sample population:

• all vendors onboarded during the period

• all high-risk vendors onboarded during the period

• all critical vendors renewed during the period

• all vendors with data access

• all vendors with open issues

Again, the right population depends on what the test is trying to conclude.

A sample chosen without a clear objective is not reliable.

Step 2: Define the population

The population is the full set of items subject to the control.

This is the most important sampling step.

A sample is only meaningful if the population is complete and correct.

Population examples:

Population definition should include:

• source system

• report name

• extraction date

• period covered

• inclusion criteria

• exclusion criteria

• owner

• completeness check

• reviewer

Many control testing failures are really population failures.

If the population is wrong, the sample is wrong.

If the sample is wrong, the conclusion is weak.

Population completeness should be tested

Population completeness should not be assumed.

Ask:

• Who generated the population?

• What system or source was used?

• Does the population include all in-scope items?

• Are exclusions documented?

• Does the population include high-risk items?

• Does it include exceptions?

• Does it include terminated users, emergency changes, renewals, privileged accounts, or other special cases?

• Does the period match the test period?

• Has the population been reviewed?

Example:

An access review population that excludes privileged users may make the sample look clean while missing the riskiest accounts.

A vendor population that excludes renewals may miss vendors with ongoing risk.

A change population that excludes emergency changes may miss the very items most likely to have exceptions.

Population quality is sampling quality.

Step 3: Confirm period and scope

Sampling depends on timing and scope.

Confirm:

• testing period

• control operating period

• evidence period

• report period

• system scope

• business process scope

• entity scope

• vendor scope

• framework scope

• audit scope

This matters because the same control may support multiple frameworks.

For SOC 2, the evidence must support the relevant report period and system scope, since SOC 2 reports on controls relevant to the trust services categories selected for the service organization.

For SOX, the sample may need to align with ICFR scope, key controls, financial reporting systems, and the timing of management assessment or external audit procedures. PCAOB AS 2201 emphasizes the role of risk assessment in selecting controls to test and determining the evidence needed for a given control.

For NIST-aligned cyber testing, the sample should preserve risk and outcome context rather than treating CSF outcomes as a checklist. NIST CSF 2.0 states that its Core outcomes are not a checklist of actions and should vary by organization and use case.

The calendar and sample design should make those differences visible.

Step 4: Assess risk and control criticality

Sample design should be risk-based.

Consider:

• control criticality

• framework impact

• financial reporting relevance

• customer assurance relevance

• regulatory relevance

• data sensitivity

• asset criticality

• vendor criticality

• prior failures

• evidence quality

• process complexity

• automation level

• volume

• frequency

• incident history

• issue history

A low-risk control with strong evidence and no prior failures may support a smaller sample or periodic testing.

A high-risk control with prior failures may require a larger sample, targeted sample, stratified sample, or full-population testing.

Risk should influence:

• whether to sample

• sample size

• sample method

• whether high-risk items are forced into the sample

• whether retesting is needed

• whether exceptions escalate

Sampling should not be one-size-fits-all.

Step 5: Decide full-population testing vs sampling

Sampling is not always the best answer.

Use full-population testing when:

• population is small

• every item is high impact

• control is high risk

• automated testing is feasible

• prior failures are frequent

• control is new or unstable

• audit or regulatory expectations require broader evidence

• the conclusion needs stronger support

• exceptions are unacceptable

Use sampling when:

• population is large

• control is routine and stable

• risk is moderate or low

• prior performance is strong

• full testing would be inefficient

• sample design can support the conclusion

• evidence quality is reliable

• framework and audit expectations allow it

Example:

Sampling should improve efficiency without weakening the conclusion.

Step 6: Choose the sampling method

Different sampling methods serve different purposes.

Random sampling

Random sampling gives each item an equal chance of selection.

Useful when:

• population is complete

• items are relatively similar

• no known risk stratification exists

• unbiased representation is important

Judgmental sampling

Judgmental sampling selects items based on tester judgment.

Useful when:

• high-risk items need review

• the tester wants to examine specific scenarios

• population is small or specialized

• known risk indicators exist

Weakness:

• may not support broad conclusions unless documented carefully

Risk-based sampling

Risk-based sampling selects more items from higher-risk groups.

Useful when:

• risk varies across the population

• certain systems, vendors, users, or transactions matter more

• control failure impact varies

Stratified sampling

Stratified sampling divides the population into groups and samples each group.

Useful when:

• population has meaningful categories

• high-risk and low-risk items should both be represented

• business units, systems, geographies, or control types differ

Targeted sample plus random sample

This combines high-risk selection with representative selection.

Useful when:

• high-risk items must be tested

• the team still wants broader population coverage

• executives or auditors need confidence in both risk areas and general operation

Example:

Test all critical changes, plus a random sample of standard changes.

This is often a practical Connected GRC approach.

Step 7: Determine sample size

Sample size should be based on risk, objective, population, control frequency, framework, prior results, and testing methodology.

There is no universal sample size that works for every control.

PCAOB AS 2315 notes that the sufficiency of evidence is related to sample design and size, among other factors, and that sample size depends on the objectives and efficiency of the sample design.

Factors that may increase sample size:

• higher risk

• key control

• prior failures

• larger population

• manual control

• complex process

• weak evidence quality

• high variability

• significant framework impact

• SOX or audit reliance

• control owner changes

• recent process change

• repeat findings

Factors that may reduce sample size or support targeted testing:

• small population

• automated control

• strong prior performance

• low risk

• stable process

• strong monitoring

• high-quality evidence

• full-population analytics available

The most important point:

Sample size should be documented.

The test record should explain why the sample size was chosen.

Sample size should not be copied blindly

Many teams use default sample sizes.

That can be helpful for consistency.

But default sample sizes become dangerous when copied without context.

For example:

• Testing 5 out of 500 low-risk items may be too weak.

• Testing 25 out of 25 high-risk items may be appropriate.

• Testing 3 vendor reviews may be weak if those vendors include critical providers.

• Testing 10 changes may be reasonable for a stable non-SOX system but weak for a high-risk financial reporting system.

• Testing a sample of AI use cases may be inappropriate if only a few high-risk AI systems exist.

A default sample size should be a starting point.

Not a substitute for judgment.

Step 8: Select and document the sample

Sample selection should be documented.

Capture:

• population source

• population size

• selection method

• sample size

• sample criteria

• selected items

• selection date

• selector

• rationale

• exclusions

• high-risk items forced into sample

• randomization method, if used

• replacement rules

This matters because reviewers, auditors, and future testers need to understand how the sample was selected.

A sample selected in a meeting and never documented is hard to defend.

A sample generated from a connected population record is much easier to defend.

Connected GRC should preserve:

• source population

• sample selection

• evidence records

• test results

• exceptions

• issue links

• conclusion

Sampling should have an audit trail.

Step 9: Test attributes and evaluate exceptions

The sample is only useful if test attributes are clear.

For each sampled item, define what is being tested.

Example: Change management sample attributes

• change request exists

• approval occurred before implementation

• testing evidence exists

• implementation date documented

• emergency change process followed, if applicable

• segregation of duties maintained

• post-implementation review completed, if required

Example: Vendor review sample attributes

• risk tier assigned

• due diligence completed before approval

• cyber review completed

• privacy review completed

• contract review completed

• open issues remediated or accepted

• approval documented

Example: Access review sample attributes

• population complete

• privileged users included

• reviewer appropriate

• review completed on time

• exceptions documented

• exceptions remediated or approved

• final signoff retained

For each exception, capture:

• item failed

• attribute failed

• severity

• root cause

• framework impact

• issue required

• remediation required

• retest required

Exceptions should not stay in the sample workbook.

They should connect to issue workflows where remediation is needed.

Step 10: Link results to issues, remediation, validation, and dashboards

Sampling results should update the broader GRC model.

A failed sample should connect to:

• control

• test result

• evidence

• affected framework

• affected risk

• issue

• remediation plan

• validation requirement

• dashboard status

If exceptions are found, ask:

• Are they isolated or systemic?

• Do they indicate control design weakness?

• Do they indicate population problem?

• Do they affect risk appetite?

• Do they affect SOX, SOC 2, ISO, NIST, internal audit, or regulatory reporting?

• Does sample result support the planned conclusion?

• Is expanded testing needed?

• Is risk acceptance needed?

• Is remediation required before reporting?

PCAOB AS 2315 highlights sampling risk: when testing is restricted to a sample, conclusions may differ from what would be reached if the procedure were applied to all items in the population.

That is why exceptions should be evaluated carefully.

One exception may be isolated.

One exception may indicate a broader control failure.

The workflow should help the team decide.

Sampling by Control Frequency

Control frequency affects sampling.

Annual controls

Examples:

• annual policy review

• annual business continuity test

• annual vendor review

• annual risk assessment

Often, full-population testing may be appropriate because there may be only one item.

If there are multiple annual controls by business unit or system, sample design should reflect scope.

Quarterly controls

Examples:

• quarterly access review

• quarterly financial review

• quarterly control owner certification

Sampling may include all quarters or selected quarters depending on risk and framework expectations.

For high-risk or SOX-relevant controls, testing multiple periods may be needed.

Monthly controls

Examples:

• monthly reconciliation

• monthly privileged access review

• monthly vulnerability review

Sampling may select months based on risk, coverage, and prior performance.

A common approach is to include selected months across the period, including higher-risk months if known.

Transaction-level controls

Examples:

• production change approvals

• access requests

• vendor onboarding

• incident records

Sampling usually focuses on a population of transactions.

Risk-based or stratified sampling is often useful.

Continuous controls

Examples:

• automated monitoring

• configuration enforcement

• continuous vulnerability scanning

Testing may focus on system configuration, exception reports, alert handling, and completeness rather than sampling individual instances.

Where full-population analytics exist, sampling may not be necessary.

Sampling by Control Type

Manual controls

Manual controls often require more careful sampling because human execution can vary.

Sampling should consider:

• performer

• reviewer

• timing

• evidence

• exception handling

• prior failures

Automated controls

Automated controls may require testing configuration, change management, access to the system, and exception handling.

Sampling may be less relevant if the automation is consistently applied, but evidence over configuration and operation still matters.

Management review controls

Management review controls often require testing precision.

Ask:

• What did management review?

• What thresholds were used?

• What exceptions were identified?

• What follow-up occurred?

• Was the review documented?

• Was the underlying report complete and accurate?

For SOX, management review control testing may require particular care because ICFR assurance depends on whether the review is precise enough to detect material misstatement risk.

IT general controls

ITGC sampling often involves access, change management, operations, and security controls.

Sampling should align with system scope, control frequency, and framework relevance.

Vendor controls

Vendor controls should sample based on vendor risk tier, data access, system access, criticality, and renewal status.

Testing only low-risk vendors creates false confidence.

AI governance controls

AI governance controls may require sampling AI use cases by risk tier.

High-risk AI use cases should often be fully tested if the population is small.

Lower-risk use cases may be sampled.

SOX Sampling Considerations

SOX sampling should be treated carefully.

Consider:

• financial reporting risk

• significant accounts and disclosures

• relevant assertions

• key control designation

• control frequency

• control precision

• population completeness

• evidence quality

• prior deficiencies

• management review controls

• ITGC dependencies

• deficiency evaluation

• retesting timing

PCAOB AS 2201 emphasizes a risk-based approach, including focusing more audit attention on higher-risk areas and selecting controls to test based on risk.

Practical SOX sampling questions:

• Is the control key?

• Is the population complete?

• Does the sample cover the period needed?

• Are higher-risk transactions included?

• Does the sample support management’s ICFR conclusion?

• What happens if one exception is found?

• Is expanded testing needed?

• Is deficiency evaluation required?

SOX sampling should not be managed casually.

SOC 2 Sampling Considerations

SOC 2 sampling should align with:

• report period

• system scope

• trust services category

• control description

• operating effectiveness

• evidence availability

• auditor expectations

• customer assurance impact

Practical SOC 2 sampling questions:

• Does the sample cover the report period?

• Does it cover in-scope systems?

• Does it test operating effectiveness?

• Does evidence support the selected trust services criteria?

• Are exceptions documented?

• Is management response available?

• Does the exception affect customer assurance?

AICPA describes SOC 2 reports as intended for users who need detailed information and assurance about service organization controls relevant to security, availability, processing integrity, confidentiality, or privacy.

That means sample conclusions should be defensible to external users, not only internal teams.

ISO 27001 Sampling Considerations

ISO 27001 sampling should consider the information security management system.

Sampling may apply to:

• risk assessments

• risk treatment actions

• control implementation evidence

• internal audit findings

• corrective actions

• policy reviews

• management reviews

• ISMS scope records

• supplier reviews

Practical ISO sampling questions:

• Does the sample represent the ISMS scope?

• Does it support control implementation evidence?

• Does it show continual improvement?

• Are nonconformities tracked?

• Are corrective actions validated?

• Does management review cover the sampled evidence?

ISO describes ISO/IEC 27001 as defining requirements an information security management system must meet.

That means sampling should not focus only on technical controls.

It should also support management-system evidence where needed.

NIST-Aligned Sampling Considerations

NIST CSF-aligned sampling should preserve risk and outcome context.

Sampling may apply to:

• asset records

• vulnerability remediation

• incident response records

• vendor cyber reviews

• monitoring alerts

• recovery tests

• access reviews

• cyber risk assessments

Practical NIST sampling questions:

• Which CSF outcome is being evaluated?

• Which internal control supports that outcome?

• Which assets, services, vendors, or data are in scope?

• Is the sample risk-based?

• Are critical assets represented?

• Are incidents and issues linked?

• Does the result inform cyber risk posture?

NIST CSF 2.0 says its Core outcomes are arranged by Function, Category, and Subcategory, and that outcomes are not a checklist of actions.

A NIST-aligned sample should support the cyber risk story.

Not just a checklist conclusion.

Internal Audit Sampling Considerations

Internal audit sampling may vary depending on audit objective.

Internal audit may use:

• risk-based sampling

• judgmental sampling

• random sampling

• targeted high-risk samples

• full-population testing

• data analytics

• stratified sampling

Internal audit sampling should document:

• audit objective

• population

• selection method

• sample rationale

• attributes tested

• exceptions

• conclusion

• finding criteria

• management response

• validation approach

Internal audit should also consider whether sample results indicate broader process weakness.

A sample exception may become a finding.

A repeat sample exception may become a root-cause theme.

Evidence Needed for Sampling

A sampling record should include evidence of both selection and testing.

Sample selection evidence

• population report

• extraction date

• inclusion criteria

• exclusion criteria

• sample method

• sample size rationale

• selected items

• randomization or selection support

• reviewer approval, where needed

Test evidence

• evidence for each sampled item

• attributes tested

• pass / fail result

• exception notes

• reviewer signoff

• issue link

• retesting evidence, if needed

Conclusion evidence

• overall test conclusion

• exception evaluation

• expanded testing decision, if applicable

• issue creation

• remediation plan

• validation status

• dashboard update

Sampling evidence is not only the sampled items.

It is also the record of how the sample was selected and evaluated.

Sampling Dashboard

A sampling dashboard should show:

Sampling should not disappear inside testing workpapers.

It should be visible where it affects assurance.

Common Sampling Mistakes

Mistake 1: Sampling before defining the population

A sample without a complete population is unreliable.

Mistake 2: Picking convenient samples

Convenience sampling creates bias and weakens conclusions.

Mistake 3: Using the same sample size for every control

Risk, frequency, volume, and framework context matter.

Mistake 4: Ignoring high-risk items

High-risk items should often be included or stratified.

Mistake 5: Reusing samples without updating scope

A sample from last cycle may not reflect current risk or population.

Mistake 6: Ignoring exceptions

One exception may be isolated or may indicate a broader failure.

Evaluate it.

Mistake 7: Not documenting sample rationale

If no one can explain the sample, assurance is weak.

Mistake 8: Sampling when full testing is better

Small or high-risk populations may require full testing.

Mistake 9: Treating sampling as evidence reuse

Sampling and evidence reuse are different.

Both need governance.

Mistake 10: Not connecting sample failures to issues

A sample exception that requires remediation should create or link to an issue.

A Practical Sampling Checklist

Use this checklist before approving a sample.

If several answers are no, the sample is not ready.

A Practical Test for Your Sampling Process

Pick one control that was sampled during the last testing cycle.

Ask whether your GRC model can quickly show:

• control objective

• population

• population source

• population completeness check

• period

• scope

• sample method

• sample size rationale

• selected items

• high-risk items included

• evidence for each item

• test attributes

• exceptions

• exception evaluation

• issue links

• remediation

• retesting

• validation

• framework impact

• dashboard status

If answering those questions requires spreadsheets, email, audit workpapers, shared drives, and meeting memory, sampling is not connected enough.

That is common.

It is also the opportunity.

Final Thought

Sampling is not a shortcut.

It is a method for building assurance efficiently.

Done poorly, it weakens confidence.

Done well, it helps teams test the right evidence, identify meaningful exceptions, reduce duplicate work, and support reliable conclusions.

A good sampling process starts with the control objective.
It defines the population.
It confirms period and scope.
It considers risk.
It decides whether sampling is appropriate.
It documents method and size.
It tests clear attributes.
It evaluates exceptions.
It links failures to issues.
It tracks remediation and validation.
It updates dashboards.

That is how to choose samples for control testing without weakening assurance.

In Connected GRC, sampling is not hidden inside a testing file.

It is part of the control assurance workflow.