Information Literacy Considerations for GenAI Usage

The role of librarians:

Two key parts of the work I do are

  1. Connecting patrons to resources that meet their information needs
  2. Supporting the critical appraisal of these resources

Today, I'd like to talk to you about 

  • The search & critical appraisal processes
  • How AI and GenAI have impacted these processes
  • What to consider when developing information literacy (finding & using information) in the current landscape

First: Searching

The Library Catalogue 

Databases

But also...
 

It depends on the information need!

  • What type of source?
  • How many sources?
  • How will this information be used?

Machine Learning (ML)

  • Ranking results in terms of relevancy
  • Related search suggestions

Natural Language Processing (NLP)

  • Interpreting queries
  • Understanding connections between words
  • How words relate to concepts

"trump news today"

"Card Catalog" by yanajenn is licensed under CC BY-SA 2.0.

Why do we care?

Bias in AI is well-documented

Twitter / @Chicken3gg

And Search Engine Optimization might impede fairness

Second: Critical Appraisal

Is this resource any good?

  • Does this make sense?
  • Are there frequent errors?
  • Is this a reputable source?
  • Can I find other sources that confirm this?

Or for academic publishing:

  • Where was this published?
  • Who funded this?
  • Do I recognize any authors or their institutions?
  • Do their choice of methods/data/participants make sense and are their conclusions logical?

While potentially unconscious, strategic searching and critical appraisal are skills you've developed.

However, this is not the case for everyone

Hallucinations are common!

An important distinction:

 

What is Retrieval-Augmented Generation - AWS

Things I'm noticing:

1. Lack of awareness about references 

ChatGPT

2. Thinking a single citation is enough

Acknowledging use ≠ Citing AI as information source

How to cite ChatGPT - APA Style Guide

How Students Are Actually Using Generative AI | Harvard Business Publishing Education

3. Using tools that can't search, for search purposes

Without Critical Appraisal, it becomes easier for GenAI users to:

  1. Believe false information to be legitimate
  2. Unconsciously adopt biases built into training data
  3. Simplify nuanced issues
  4. Miss out on alternative voices or perspectives 

Select one of the following tools:

  • Perplexity

  • Consensus

  • ChatGPT

Consensus 

Consensus DOES provide sources, however, we don't have a sense of why the 10-20 results were selected (citations, content, other), and the generated content appears to only be based on abstracts (which don't contain the nuance of the full text articles)

Ovid Medline

(creatine and ("muscle build*" or "muscle grow*")).ab,ti.

Medline (and other databases) will present ALL results that meet the criteria of your search query, so you're less likely to miss information

  • Look for information on:
    1. Whether the tool can search the internet or another resource besides training data
    2. Which types of resources are being searched (academic texts VS social media VS other sources)
    3. How results are ranked (number of citations, relevance, other)
    4. Which sources are used to generate text (if any)
  • Ask a question or prompt that you feel you already understand well and evaluate:
    1. How well the tool addresses your question
    2. Whether the sources are reputable
Chatbots RAG AI Databases
Natural Language
Sources
 
Transparent Results
Chatbots RAG AI Databases
Natural Language
Sources
 
Transparent
Results

increasing ease of critical appraisal 

decreasing ease of use

 Databases > Search Engines/RAG AI > Chatbots

I am not saying:

I am saying:

 Databases ≠ Search Engines / RAG AI ≠ Chatbots

Systematic Approach to Information Gathering

"Evidence-based Medicine (EBM)" by Tingjoh is licensed under CC BY-SA 4.0.

High value is placed on:

  • transparency
  • reproducibility 
  • accuracy
  • comprehensiveness

Therefore, article databases are utilized

Takeaways:

  • Choose the right tool
  • Using a sophisticated search tool requires an investment of time to learn
  • Sources matter; having access to full-text resources makes critical appraisal easier
  • An understanding of why results are generated can also help control bias
  • Critical appraisal is a skill that needs to be learned and practiced

Evaluating AI Information Literacy

Searching:

  • Are my students able to identify appropriate use cases for Generative AI that still involve them learning?
  • Can they pick an appropriate tool depending on the task, or are they consistently using a single tool regardless of the task?
  • Do they understand the difference between acknowledging usage of GenAI, and citing GenAI as an information source

Critical appraisal:

  • Are my students aware of issues around bias and lack of transparency in AI?
  • Do they have the critical thinking skills to appraise Generative AI outputs, including identifying hallucinations and misinterpretations of original content?
  • Are they able to find the full version of infomation cited by AI to evaluate?

What we're doing

  • Talking to students!
  • Testing out tools and putting together resources for students
  • Looking for opportunities to collaborate

Thank you!

contact: catherine.gracey@unb.ca