Assessing the Academic Value of Sources
The source opens the field wide — literature is everything written on a topic — and then narrows it in the next sentence to what is credible and trustworthy. Both moves matter, and they are two separate filters.
What the source counts as literature
Two sentences, two different jobs. The first is deliberately inclusive: seven categories are named and the list ends with etc, which means the boundary is not the format but the relation to your topic. The second is a weighting instruction — predominantly, not exclusively — that tells you where the bulk of your review should come from without ruling anything out.
That inclusiveness is doing real work for a project manager. Much of what bears on a delivery question is written down in forms that are not journal articles: government reports, theses, technical and regulatory material, industry records. Under this definition all of it is literature. What it is not is automatically academic, and the second sentence is what stops the review filling up with it.
Relevant does not mean identical
Relevance is a relation between a source and your topic, not a property the source carries around with it. The same article is relevant to one project and irrelevant to another, which is why nobody can hand you a reading list and why the judgement cannot be delegated to a database.
The commonest way this judgement is made badly is by setting it too tight. The supplied material reports a recurring pattern: researchers conclude there is nothing in the literature on their topic, and in virtually every case the reason is that they looked only for work that was exactly the same as their own. That failure and its remedy are treated on why review the literature early.
Four ways a source can relate to your topic
Only after a source has passed relevance does credibility become worth your time. Running eight quality checks over an item that is not about your question is wasted effort, and the reverse — collecting relevant material without checking what it rests on — is how weak claims get laundered into your own argument.
The eight credibility questions
The source supplies these as a list to ask yourself when reviewing articles. It attributes them to a university learning guide published in 2011; the institution is not named in this library.
Running the eight over a source
The questions are posed without commentary, which leaves you to work out where to look and what a satisfactory answer actually looks like. The table supplies both. Column one is the source's; columns two and three are synthesis added here.
THE EIGHT QUESTIONS AS A WORKING CHECK
| The question | Where you look | What a pass looks like |
|---|---|---|
| 1. Peer reviewed or refereed? | The outlet's own statement about its review process, and the publication-type filter in the database you searched | The outlet states that its articles are peer reviewed, and the item is a research article rather than an editorial, a book review or a comment piece carried by the same outlet |
| 2. Reference list with matching in-text references? | The end of the item, then any two pages of body text | Every in-text citation resolves to an entry in the list, and the list is not padded with works that are never cited. This is the one question that inspects the argument rather than its packaging |
| 3. Who is the author? | The byline, the author note, the affiliation line, the acknowledgements | A named author with a stated affiliation — the source names a university, a research institute, or a government department such as a national statistics agency |
| 4. Does the text look academic? | The page or document around the text, including anything sold alongside it | The source's own test: any advertising is for academic products such as books or dictionaries rather than unrelated commercial content |
| 5. Is it written in formal language? | Any two paragraphs, chosen at random rather than from the introduction | Claims are attributed and qualified rather than asserted; where the first person appears it is describing what the researcher did, not what the writer feels |
| 6. Does it use paragraphs and headings? | The shape of the document at a glance, before reading a word | A structure you can navigate, in which the headings tell you what each part contains — the visible sign that the argument was organised before it was written |
| 7. Who is the publisher? | The title page, the imprint page, the footer of a web page | An academic publisher — the source's example is a university press. For a report, the issuing body is the publisher and should be identifiable by name |
| 8. For a website: author and copyright date? | The top and the bottom of the page, and the About section | A named author and a copyright date, which the source notes is often at the foot of the page. An organisation may stand as the author where no individual is named |
Column one reproduces the source's questions in short form. Columns two and three are synthesis; the source poses the questions without stating where to look or what a satisfactory answer is.
Do the cheap checks first
The eight are printed as a flat list, but they cost wildly different amounts of your attention. Ordering them by cost turns the list into a triage sequence you can run over a large marked set without losing a day.
A three-pass sequence over a set of marked citations
Pass one — seconds per item, from what is already on screen
Questions 4, 6 and 8. Does it look academic, does it have paragraphs and headings, and for a web item is there an author and a copyright date. These are visible before you open anything and they clear out the obvious. Nothing survives this pass on merit; things only fail it.
Pass two — a minute per item, inside the document
Questions 2, 3, 5 and 7. Reference list and matching in-text citations, named author with an affiliation, formal language, identifiable publisher. Question 2 does the most work here — it is the only one of the eight that tells you whether the argument is anchored to anything outside itself.
Pass three — outside the document
Question 1. Whether the outlet is peer reviewed cannot be answered from the article alone; you check the outlet's own statement or filter for it in the database. Leave it last because it costs the most and only matters for items that survived the first two passes.
That ordering is synthesis, not the source's. The source presents the eight in a fixed order with peer review first, which is where the weight belongs conceptually — see peer review and blind review — but not where it belongs procedurally when you are working through fifty marked citations from a database search.
What a failed question actually means
The source gives no rule for handling a no. Treating any single failure as disqualifying would exclude most government reports, every thesis and the whole of the standards literature — all of which the same source counts as literature. The failures are not equivalent, and they do not all mean the same thing.
How to act on each kind of failure
Where the eight questions run out
This list is a competent first screen and it is not a quality appraisal. Several things you would want to know about a source are simply not asked.
The last of those is the important boundary. These eight questions decide whether a source is worth reading properly. A different set of questions decides whether what you then read belongs in your review — that set is on evaluating an article before you cite it, and it is the one that asks about theoretical basis, methodology, data and conclusions.
Everything that survives both filters has to be captured properly at the moment it survives, with full bibliographic details rather than a link. That discipline is on note-taking and bibliographic records, and the search that produced the candidates in the first place is on database search technique.
What to carry forward
- Literature is everything relevant written on the topic. The format is not the boundary; the relation to your question is.
- Relevance first, credibility second. A credible source that does not bear on your question is not a source, it is a distraction.
- Relevant does not mean identical. Work from another industry, or on another question using your method, is squarely relevant.
- Of the eight questions, provenance and referencing carry the weight. Presentation is the easiest thing to imitate and the least informative.
- A single no is not automatically fatal. Government reports and theses fail the peer review question by definition and remain legitimate literature.
- The eight ask nothing about currency, funding, sample or method. Those come later, from a different set of questions.
Frequently asked questions
Does the source say I can only use peer reviewed articles?
No. It says to rely predominantly on academic sources and that credible and trustworthy articles are usually peer reviewed. Its own definition of literature explicitly includes government reports, historical records, theses and newspaper articles, none of which are peer reviewed in the journal sense. The instruction is a weighting, not an exclusion.
How many of the eight questions does a source have to pass?
The source sets no threshold and offers no scoring scheme. Practically, the questions about provenance and referencing are the ones that should decide the matter — who wrote it, who published it, what review it went through and what it cites. A source failing only the presentation questions is unusual; a source failing the referencing question should not be carrying claims in your review.
How do I check whether a journal is peer reviewed?
The source's own answer is that journals and conference proceedings will usually state whether their articles are peer reviewed, so look for that statement in the outlet's own information. Database publication-type filters are the other route, since the source separately suggests restricting a search by publication type when a result set is too large.
Can I use a website in a master's-level literature review?
The source does not forbid it and supplies a specific question for the case: is there an author and a copyright date, often at the bottom of the page. It also asks whether the text looks academic, using advertising as the test. A website that names an accountable author or issuing body, carries a date and cites its own sources can be used; one that does none of those is not literature you can build on.
What if there is very little credible literature on my topic?
That is a finding about your topic, not only about your search. The teaching material's position at the idea stage is blunt: if there is little literature on the topic you should seek advice, and it is likely you will be advised against continuing. Before accepting that conclusion, check that you have not been searching only for work identical to your own question.
Does an old article automatically fail?
None of the eight questions asks about the age of a work, so on this list a date is not a criterion at all. Currency is raised elsewhere in the same material — whether the topic has been written about recently, and how current your information needs to be — so treat currency as a separate judgement you make about the body of literature as a whole, not as a pass or fail on individual items.
References and source attribution
- The supplied teaching source: master's-level teaching material on reviewing the literature, supplied as study notes and accompanying slides. The definition of literature and the eight credibility questions are quoted from that material.
- A university learning guide, 2011 — the work the supplied source credits with the eight credibility questions. The institution is not named in this library.
- Trochim, W. M. K. 2006, Research Methods Knowledge Base — reproduced in the supplied source, and the origin of its guidance on concentrating on credible research journals.
- O'Leary, Z. 2017, The Essential Guide to Doing Your Research Project, 3rd ed., Sage Publications, London (listed among the source's references).
- Bryman, A. 2016, Social Research Methods, 5th ed., Oxford University Press, Oxford (listed among the source's references).
Suggested questions for Ask KEVOS
- Run the eight credibility questions over a source I am considering and tell me which ones it fails.
- Is a government report legitimate literature for a master's level review?
- How do I decide whether a source from another industry is relevant to my project?
- What should I do with a relevant source that has no reference list?
- Which of the eight questions matter most, and why does the source not say?