Survey Design Across Four Examined Theses
Four questionnaires, four achieved samples, and one field log nobody kept. What each of these works records about its own survey — and what all four leave out — amounts to a working specification for the one you are about to write.
Four surveys, side by side
These are four examined master's works, referred to here as Thesis G, Thesis J, Work K and the dataset thesis — the last of which arrived with its instrument, its consent sheet, its raw response file and its analysis workbook. Every figure below belongs to the study that produced it. One project is not a population, four projects are not a population either, and the defects recorded are defects in documents rather than in researchers: all four were examined and passed.
THE FIVE THINGS A READER NEEDS, FOR EACH SURVEY
| Thesis G | Thesis J | Work K | The dataset thesis | |
|---|---|---|---|---|
| Instrument length | 15 numbered items; claimed as "30 questions" | 13 short questions | At least 14 closed item groups plus 2 free-text — reconstructed, not shown | 8 questions, 37 response options |
| Response formats | Single-choice, one multi-select, a 16-statement Likert matrix, one free text | Closed multiple-choice and open-ended | Recoverable for one item group only | Single-choice radio on every question; no free text |
| Stated burden | ≈15 minutes | "less than 10 minutes" | Not stated | Not stated |
| Recruitment | "Web-based" is the whole of it | 60 invited; how they were selected not stated | "Individuals who met the selection criteria were approached" | Described as random in the thesis and as referral in the information sheet |
| Field window | Not stated | Two weeks | Not stated | Not stated |
| Achieved | 100 | 46 (plus 2 interviews) | 17 | 50 |
| Reported rate | None — no denominator | 77.4% — and it is not the survey's rate | None — no denominator | None — the words *response rate* and *invited* do not appear |
| Instrument displayed | Yes, reproduced in the thesis | Yes, plus an interview schedule | No — the work has no appendices | Yes, as issued — the only one |
All figures are each study's own. None is a benchmark, a target or a norm.
Instrument length, and what counts as a question
Thesis G describes "a questionnaire of 30 questions, with the majority using Likert-scale". Its printed instrument has fifteen numbered items: a consent disclaimer, twelve single-choice items, a matrix of sixteen Likert statements, and one free-text box. 13 + 16 + 1 = 30. The claim is exactly correct — and only if the sixteen matrix statements are counted individually and the consent disclaimer is not counted at all.
At the other end, the dataset thesis fielded eight questions — four demographic, four substantive — with thirty-seven response options between them, every question compulsory. Work K's length cannot be established at all: its instrument is never printed, never summarised item by item, and never described as to length or completion time. Its content is recoverable only from thirteen result-figure captions and the prose around them, which imply at least fourteen closed item groups and two free-text questions.
Response formats, and where they cost you
Four format decisions and their consequences
A sixteen-statement Likert matrix
Nine of the sixteen statements are double-barrelled or contain an absolute — a plan and a culture, rules that are easy to understand and follow, procedures followed at all times by all employees. A respondent who agrees with one half and not the other has no available answer, and several of these items are reported as single findings.
Mixed directions, no stated reverse scoring
Most statements are positively worded and at least two are negative. A composite index averages all of them on a fixed scale. If the negative items were not reversed before averaging, the index is internally inconsistent — and the thesis does not say whether they were.
A scale that is not monotonic
One item's options run "Must be implemented, Very Important, Fairly Important, Important, Not Important". "Important" sits below "Fairly Important", so the scale has no consistent direction and two middle responses cannot be ordered against one another.
Compulsory, with no way out
Eight of eight questions are required, with no "prefer not to say", no "other", no free text and no "none of these" on an item asking respondents to pick among five named models. A working draft of the questionnaire offered a skip on that item; the form as issued marked it required, and all fifty responses answered it.
Two format faults recur across the set and are worth checking in any instrument. Band boundaries that overlap: one instrument's experience and enterprise-size bands read 0–5, 5–10, 10–20 and <10, 10–50, 50–100, so a respondent at exactly 5, 10 or 50 falls in two categories. And conditions that point at the wrong item: one questionnaire's multi-select is conditioned on the item above the one it logically depends on. See structured and unstructured questions and question wording for the underlying rules.
Recruitment: the common gap
This is the weakest area across all four, and it is the one that determines what any of the samples can be generalised to. Thesis G's account of how a hundred people reached its questionnaire is the single word "web-based". Work K's is two sentences: individuals meeting the criteria were approached, and subsequently seventeen completed it.
Six things missing across the set, in varying combinations
- The sampling frame — no list, no register, no professional body, no employer
- The approach channel — no invitation text, no distribution method
- The invitation count — so no response rate can be computed
- The field window — only one of the four states one (two weeks)
- The platform — stated by two of the four
- A named sampling technique — the words purposive, convenience, snowball, stratified and saturation appear in none of them
One eligibility practice in the set is worth copying outright. Work K states two operational, checkable criteria — employment status and responsibility for delivery of the relevant projects — and then defends the second against a narrow reading, with a cited list of the roles it is meant to include. It is the only worked eligibility defence in this library, and nothing in the same work contradicts it. Thesis G shows the opposite: a stated eligibility threshold of more than six months' experience, an instrument whose first option is "0–6 months" with no filter applied, and a findings section that then reports on that group.
Achieved numbers, and the one reported rate
Thesis J is the only one of the four whose denominator is on the page: the survey went to 60 people and 46 answered. That makes it the only response rate in this library that can be checked. It does not survive the check.
The survey went out to 60 people and only two people were approached for interviews, making the overall response rate 77.4%.
The three others report no rate at all, and in each case the reason is the same: the numerator is stated and the denominator is not. One survey "was sent out to many" professionals. One was "web-based". One went out as an email link to an unstated number of people over an unstated period. A response rate is not an optional extra you can add later — it is the invitation count, recorded at the time. See questionnaires and response rates and non-response and sampling bias.
What each one got right
CREDIT WHERE IT IS SPECIFICALLY DUE
| Work | What it does that the others do not |
|---|---|
| Thesis G | A complete consent apparatus with eleven distinct elements — voluntariness, task, duration, retention period, secure storage, de-identification, right to withdraw, the limit on withdrawal after submission and its reason, an approval statement, an independent contact and an offer of findings. Its withdrawal clause is unusually honest: it refuses withdrawal after submission because anonymous collection makes retrieval impossible. It also prints 75 free-text responses verbatim — the largest block of primary qualitative data here |
| Thesis J | States its denominator, its field window and its completion time; shows both its instruments; states that its qualitative data were recorded and transcribed — the only work here to say so; and describes its quantitative analysis in one honest sentence, claiming no statistics beyond percentages and performing none |
| Work K | A sensitivity analysis: a third of respondents had under twelve months' exposure, so those responses were removed, the analysis was rerun on the remaining eleven, and the full cross-tabulation was printed. Every weighted average in it recomputes exactly, and the conclusion was reported as unchanged. The work never calls it a sensitivity analysis; it is the best quantitative practice in its batch |
| The dataset thesis | The instrument as issued — not a reproduction inside the thesis, but the form itself, with every response option, alongside a separate working version of the same questionnaire. The two differ in five demonstrable ways, and having both is what makes the difference visible |
Designing yours
The order these four suggest
Write the eligibility criteria as tests
Each one should be answerable yes or no about a named person. Then check that an item on your instrument can actually establish each criterion — one of these four states a criterion its own questionnaire cannot test.
Build the frame before the instrument
Name the list you will draw from, the channel you will use and the number you will approach. The invitation count is the one datum that cannot be reconstructed afterwards.
Draft items, then attack the connectives
Every and in a statement is a potential double-barrel; every all, always and never is an absolute that forces disagreement. Nine of sixteen statements in one instrument here carry one or the other.
Fix the scales
One direction, evenly spaced labels, no overlapping numeric bands, and an exit — "not applicable" or "prefer not to say" — wherever a respondent might genuinely have no answer.
Pilot it
None of these four did. Five colleagues and thirty minutes would have caught the non-monotonic scale, the overlapping bands, the misdirected condition and most of the double-barrels.
Record the field log as you go
Invitations sent, date opened, date closed, reminders, responses received, responses discarded. Every one of the four gaps above is a line in a log nobody kept.
Report length, burden and format together
State the number of items, the counting convention, the estimated completion time and the response formats used, in one place — then display the instrument.
What to carry forward
- Three of four surveys here cannot report a rate because nobody recorded the invitation count. Record it on the day you send.
- The one printed rate blends two interviewees into both halves of the fraction. Report a rate per instrument, over that instrument's own invitees.
- State the convention behind your instrument length. Thirty questions, fifteen items and thirty-one elements were all true of one questionnaire.
- An instrument that is not displayed makes every number in the study unverifiable — not wrong, unverifiable.
- Small n makes every percentage traceable to a count. Check that yours are attainable from your base before you print them.
- Each of these four does one thing better than the rest: a sensitivity analysis, an eleven-element consent text, a transcription statement, and a displayed instrument as issued.
Frequently asked questions
How do I calculate a response rate correctly?
Divide the number of usable responses to one instrument by the number of people invited to that instrument, and print both figures beside the percentage. The one checkable rate in this material — 77.4% — was produced by adding two interviewees to both the numerator and the denominator of a survey rate. The survey's own rate was 46 of 60, or 76.7%, and it appears nowhere in the document.
What should I record while my survey is in the field?
Invitations sent and to whom, the date the survey opened and closed, any reminders, responses received and responses discarded. Three of the four surveys compared here can report no response rate, and in every case the reason is the same: the numerator was recorded and the denominator was not. It cannot be reconstructed afterwards.
How many questions should a survey have?
There is no number to quote here. The four instruments run from eight questions to a claimed thirty, with stated completion times of about fifteen minutes and under ten minutes where they are stated at all. What matters more is that you state the number, the counting convention behind it and the estimated burden, and then display the instrument.
Is it a problem if my instrument is not in an appendix?
It means no reader can check your item wording, response options, ordering or filters, so your percentages have to be taken on trust. One of the four works here has no appendices at all and reports percentages and a weighted average from a questionnaire nobody can see. Its results are not thereby wrong — they are unverifiable, which is a distinct and more precise criticism.
What is a double-barrelled question and how do I spot one?
It asks about two things at once, so a respondent who agrees with one half and not the other has no honest answer. Search your draft for and inside a statement, and for absolutes like all, always and at all times. Nine of sixteen Likert statements in one of these instruments carry one or the other, and several are reported as single findings.
Do I need to pilot my questionnaire?
None of these four reports one, and the visible consequences are a scale with no consistent direction, overlapping numeric bands, a condition pointing at the wrong item, and nine compound statements. A pilot with a handful of people finds all of that in an afternoon, and it is the cheapest quality control available to a research project.
References and source attribution
- Two examined master's theses supplied as student work, both running online questionnaires: one of 100 respondents with a fifteen-item instrument including a sixteen-statement Likert matrix and 75 verbatim free-text responses, and one of 46 respondents from 60 invitations plus two interviews, both instruments displayed. Researchers, supervisors, institutions, employers and places scrubbed.
- A third examined master's work combining a systematic review with a questionnaire of 17 respondents, whose instrument is not displayed and whose content is recoverable only from its figure captions and prose. Its sensitivity analysis and its ten reported percentages were recomputed for this library against a base of 17.
- A complete research dataset supplied as student work: an eight-question survey form as issued, a separate working version of the same questionnaire, a participant information sheet, and a raw response file containing 50 complete responses, against which the achieved sample was verified.
- The supplied teaching source: weekly study notes, slide decks and assessment activities for a master's-level research methods subject in project management, which names the pilot study four times and teaches it nowhere, and warns against expecting a high response rate without quantifying one. Author, institution and year not stated in the supplied files.
Suggested questions for Ask KEVOS
- How do I work out and report a response rate for my survey?
- What should I record in a field log while my survey is open?
- How do I check that my reported percentages are attainable from my sample size?
- What belongs in a survey appendix so my results can be checked?
- How do I fix double-barrelled and absolute statements in a Likert matrix?
- How should I describe recruitment when I used referrals rather than a list?