Survey protocol and analysis plan | Source-led method research (E1+E3) | Checked
Quick answer
This version cannot say which evidence or product attributes influence learners because no learner has been recruited and no response has been collected. It instead defines a reproducible way to ask that question without turning synthetic fixtures, convenience samples, or platform marketing into survey findings.
The planned study would compare 12 evidence attributes across four technical-learning subject cells, then report sample-specific counts, shares, and medians. It would not claim a representative U.S. estimate, a causal effect, a provider winner, or a conventional margin of sampling error from the planned nonprobability sample.
Evidence and scope
- What this version contains
- A pre-specified survey protocol, draft instrument, response schema, synthetic validation fixtures, and analysis rules.
- What it does not contain
- No recruitment, participants, responses, field dates, incentives, response rate, findings, population estimate, or provider ranking.
- Planned population
- U.S. adults age 18 or older who considered, started, or paid for technical learning during the preceding 12 months.
- Authorization state
- Fielding, human-participant research, contact collection, incentives, and publication are blocked.
- Main limitation
- A transparent method cannot answer the buyer question until a later, separately authorized study recruits real participants and reports its achieved sample.
The protocol follows a disclosure-first approach: define the population, instrument, recruitment method, sample type, processing rules, privacy controls, limitations, and reporting boundaries before collecting data. That approach is consistent with current AAPOR ethics and disclosure standards and U.S. Census Bureau transparency guidance. It is not an endorsement, certification, legal determination, or human-subjects approval.
Protocol status and stopping rules
- Method version: 1.0.0; fingerprinted against the exact machine-readable protocol.
- Planned target: 600 qualified completes, with 150 in each of four subject cells, after separate fielding authorization.
- Current achieved sample: zero participants and zero responses.
- Sampling claim: planned nonprobability quotas; no claim that every eligible U.S. learner has a known selection probability.
- Reporting rule: no subgroup result below unweighted n=50; no population precision claim; no provider ranking.
- Data rule: no names, email addresses, phone numbers, exact locations, employer or school names, account identifiers, salary, health information, disability status, credentials, tokens, or secrets.
- Fielding remains blocked until governance, consent language, recruitment source, processor/storage, deletion, incident, incentive, accessibility, and instrument-version controls are approved.
Because fielding has not started, AAPOR response-rate definitions are not yet applicable. A later report would need actual disposition records and a declared rate formula rather than a guessed denominator. See the AAPOR Standard Definitions and Response Rates Calculator.
The decision this survey is designed to study
Course buyers rarely make one generic “platform” decision. They may be choosing a free starting resource, a single paid course, a multi-course certificate, a lab environment, or a recurring subscription. The same learner can value prerequisites, hands-on practice, billing clarity, credential meaning, or work portability differently depending on that job.
The protocol therefore starts with one recent decision, one primary subject, one goal, and one learning format. It asks which evidence the respondent actually used, which evidence was difficult to verify, which attributes were must-have conditions, and which one mattered most to the final decision. It does not ask respondents to score named providers, repeat marketing slogans, or predict their future salary.
Planned population and eligibility
The intended population is U.S. adults age 18 or older who considered, started, or paid for technical learning in the prior 12 months. That definition sets a recent decision window and excludes minors. It does not create a sampling frame. The exact panel, list, recruitment route, coverage, incentives, and duplicate-prevention method remain fielding blockers and would have to be disclosed. The distinction between a target population and an achieved sample is central to Census sample-design guidance and AAPOR disclosure standards.
Consent and governance gate
A later fielding version would require truthful study purpose, sponsorship, voluntary participation, expected duration, data use, foreseeable risks, withdrawal language, contact route, and any incentive terms. This package does not decide whether a particular regulatory review, exemption, or consent process applies. Those decisions must be made before recruitment under the applicable governance process, informed by the HHS Office for Human Research Protections regulations index and informed-consent guidance.
What the instrument would measure
The instrument separates evidence attributes that are often collapsed into a vague “quality” label. Each attribute can be rated for importance, selected as a must-have condition, or identified as the single most decisive factor. This protocol is not a ranking: it records the respondent’s decision evidence within a declared context.
On narrow screens, scroll horizontally to view every column.
| Attribute | What the question is intended to distinguish | What it cannot prove |
|---|---|---|
| Goal and prerequisite fit | Whether the offering matches the stated capability, credential, and starting level. | Teaching effectiveness or readiness. |
| Curriculum and freshness | Whether scope and maintenance evidence were visible before the decision. | Technical accuracy of every lesson. |
| Practice and feedback | Whether labs, projects, assessment, and feedback were material to the choice. | Lab reliability or feedback quality. |
| Support and credential | Whether human support or a specific credential route affected the decision. | Credential recognition or career outcomes. |
| Access and cost | Whether access duration, ownership, price, renewal, cancellation, or refund terms mattered. | A complete checkout total or future renewal amount. |
| Device, accessibility, and portability | Whether usable devices, captions, transcripts, accessibility needs, or retained work affected the choice. | Accessibility conformance or lawful reuse of every artifact. |
| Independent evidence | Whether transparent methods, independent records, or credible peers informed the decision. | That popularity equals quality. |
The questionnaire contains 22 planned items across consent and eligibility, decision context, evidence used, attribute tradeoffs, decision outcome, and optional broad sample-description fields. Question wording, order, response options, skip logic, and interface behavior must be tested before fielding. The current instrument is a versioned draft, not a deployed form; the design boundary draws on Census questionnaire-evaluation resources and the W3C forms tutorial.
How later results would be reported
No result chart is shown because there are no responses. The following table is a reporting contract, not a placeholder for invented numbers.
On narrow screens, scroll horizontally to view every column.
| Output | Permitted calculation | Release condition |
|---|---|---|
| Achieved sample | Unweighted counts by subject, decision stage, goal, and format. | Disclose recruitment source, field dates, exclusions, and completed interviews. |
| Evidence used | Unweighted count and share selecting each evidence source. | Denominator and multi-response rule visible. |
| Attribute importance | Full five-category distribution; optional sample median. | No invented composite score or provider attribution. |
| Must-have and decisive attributes | Unweighted count and share. | Report “none” explicitly and retain missingness. |
| Cross-tabs | Sample-specific descriptive comparison. | Unweighted subgroup n at least 50; otherwise mark insufficient. |
| Open text | Future codebook-based thematic counts after human review. | Separate coding protocol, privacy review, coder count, and reliability record. |
The planned nonprobability sample would not receive a conventional probability-sample margin of error. Any weighting would require a separate specification, source totals, diagnostics, and revised analysis approval. A later report must state whether data were unweighted, how quality checks were applied, which records were excluded, and what uncertainty remains, following AAPOR’s current disclosure requirements.
Unknowns, exclusions, and fielding blockers
- The recruitment source, sample supplier, panel coverage, incentive, duplicate-prevention method, and likely participation rate are unknown.
- No cognitive interview, pilot, accessibility audit, translation review, mobile test, or timing study has been completed.
- No processor, storage environment, retention period, deletion workflow, access roster, breach-response route, or approved public-use dataset has been authorized.
- The proposed quotas balance subject cells for comparison; they do not make the sample representative of U.S. technical learners.
- Self-reports may be affected by recall, interpretation, satisficing, social desirability, and nonresponse.
- Optional open text can introduce identifying or sensitive information and therefore remains disabled until a separate coding and redaction process is approved.
- The survey can describe reported decision evidence. It cannot establish course quality, causal influence, labor-market value, credential recognition, or the best provider.
Privacy controls must cover the whole lifecycle, not just the consent screen. The protocol minimizes fields and prohibits direct identifiers, but any later collection still needs approved storage, access, retention, deletion, sharing, and de-identification controls. The boundary is informed by the NIST Privacy Framework, NIST de-identification guidance, and FTC data-security guidance.
Methodology and version control
Versioned method
Protocol 1.0.0 is stored as machine-readable JSON with an exact instrument, controlled attribute registry, planned sample rules, privacy prohibitions, analysis rules, and stopping conditions. Its fingerprint is validated before packaging. A separate response schema rejects missing required fields, out-of-range scales, unknown options, incomplete attribute matrices, direct identifiers, and a response that fails eligibility.
Synthetic validation is not survey evidence
Eight valid and eight invalid synthetic fixtures test the reference analyzer and schema. They exist only to prove deterministic behavior. They are labeled synthetic, are not participants, are not included in the achieved sample, and must never appear in findings. This separation follows AAPOR’s current distinction between human participants and AI-generated or modeled responses.
Required future version change
Fielding would require a new version with explicit authorization, finalized recruitment and consent records, data-processing and retention controls, accessibility and pilot results, incentive disclosure, field dates, and an accountable owner. Findings would require another version containing achieved sample, dispositions, exclusions, response or participation-rate definition, weighting status, processing steps, limitations, and correction/version history.
Commercial and editorial independence
No affiliate relationship, advertising placement, enrollment link, provider payment, or commercial CTA is part of the protocol. Provider names are not asked in the core instrument, and no provider can become a winner through the analysis.
See How we test for the site-wide evidence labels and testing boundary used by this protocol.
Sources
- Code of Professional Ethics and Practices – AAPOR; checked . Professional survey-research ethics and disclosure guidance; not a legal determination or study approval.
- Best Practices for Survey Research – AAPOR; checked . General survey-design practice; not evidence that this protocol has been fielded.
- Disclosure Standards – AAPOR; checked . Method-disclosure framework; no results exist in this package.
- Transparency Initiative – AAPOR; checked . Transparency expectations; does not certify this protocol.
- Standard Definitions – AAPOR; checked . Disposition and rate definitions apply only after real fielding records exist.
- Response Rates Calculator – AAPOR; checked . Rate-calculation resource; response rate is not available before recruitment and fielding.
- Condemned Survey Practices – AAPOR; checked . Ethics boundary against misleading or sales-disguised research.
- Task Force on Responsible AI Integration in Survey Research Report – AAPOR; checked . AI-use disclosure and validation context; this package uses no synthetic responses as findings.
- Regulations – HHS Office for Human Research Protections; checked . Federal human-research regulation reference; the package makes no IRB, exemption, or legal determination.
- Coded Private Information or Biospecimens Used in Research – HHS Office for Human Research Protections; checked . Coded-data governance context; not directly applicable to a survey that has not collected data.
- Informed Consent FAQs – HHS Office for Human Research Protections; checked . Consent-reference context; exact applicability requires an authorized governance determination before fielding.
- Informed Consent Checklist – HHS Office for Human Research Protections; checked . Checklist reference; no consent form has been deployed.
- Data Collection Instrument and Communication Evaluation – U.S. Census Bureau; checked . Questionnaire evaluation context; no Census review of this instrument occurred.
- Statistical Quality Standards – U.S. Census Bureau; checked . Government statistical-quality framework; this is not an official-statistics product.
- Statistical Quality Standard A3: Developing and Implementing a Sample Design – U.S. Census Bureau; checked . Sample-design principles; planned nonprobability quotas do not support population precision claims.
- Statistical Quality Standard F2: Providing Documentation to Support Transparency in Information Products – U.S. Census Bureau; checked . Documentation framework; no survey findings are released.
- NIST Privacy Framework, Version 1.0 – NIST; checked . Privacy-risk framework; not a privacy certification or compliance determination.
- De-Identifying Government Datasets: Techniques and Governance – NIST; checked . De-identification governance context; de-identification does not eliminate every re-identification risk.
- Forms Tutorial – W3C Web Accessibility Initiative; checked . Accessible-form design guidance; no deployed survey interface was audited.
- Protecting Personal Information: A Guide for Business – Federal Trade Commission; checked . Data-minimization and security guidance; not individualized legal advice.