Ryan and Debi & Toren

Qualitative Analysis Using LLMs on a Local Machine

rcragun

While most of my research is quantitatively oriented, I am a mixed-methods researcher and periodically use qualitative sociological methods for research projects (typically interviews). I have used a variety of software packages to analyze qualitative data over the years, from NVIVO to Taguette to an abandoned R package to just coding text in word processing software. My wife and I recently had the idea of using ChatGPT to analyze qualitative data and it worked fairly well. We have a paper under review about that.

Some people have raised ethical concerns with using ChatGPT, specifically that OpenAI, the company behind ChatGPT, will have a copy of the data, the data will be used to improve their models, and the information could be leaked. While I don’t think any of these issues are serious concerns if your qualitative data is carefully de-identified and anonymized, I have spent some time trying to do qualitative analysis on my personal computer. I have a custom-built computer that runs Linux (KDE 24.04 at present). I have an NVIDIA GeForce RTX 3090 GPU with 24 gigabytes of RAM that I can use to run these models. This means I can run small and medium-sized models, but not the largest models. I was able to get Ollama installed on my machine (see here). I tested this with five models: llama3(8B), llama3-gradient (8B), command-r (35B), mixtral (47B), and wizardlm2 (8B). I have been testing whether locally installed LLMs can analyze qualitative data in a fashion comparable to ChatGPT. As of 6/21/2024, the answer is… meh. Not really.

This post documents my efforts and the results. For all of the analyses below, I am using this collection of six articles from Christianity Today (January 2007) that I used for some analyses I did more than a decade ago. I’m not using any of my interview data as I am still analyzing it and do not want to make that public.

Test 1 – Recognize Contents of PDF

I started with a very simple test – can the LLM analyze the contents of the document and tell me what is inside it? With all of the models, I used the following prompt:

The attached document contains six articles from Christianity Today. Please list the titles of the articles and the page numbers where the articles begin. If possible, please present the results as a table.

Here are the results by model:

ChatGPT 4o

Here is a table listing the titles of the articles from Christianity Today and the page numbers where the articles begin:

Article TitlePage Number
Sex Isn’t Work2
Child Sex Tours3
Red-Light Rescue4
Reviewing the Fundamentals9
The Pain at New Life10
Devastated by an Affair12

Let me know if you need further assistance with this document.

command-r

Sure! Here’s a table with the titles of the articles from Christianity Today and their corresponding starting page numbers:

Article TitlePage Number
Sex Isn’t Work2
Child Sex Tours3
Red-Light Rescue4
Reviewing the Fundamentals9
The Pain at New Life10
Devastated by an Affair12

llama3-gradient

Here are the answers to your questions.

Table of Contents

Sex Isn’ t Work … 2
Child Sex Tours 3
Red- Light Rescue 4
Reviewing the Fundamentals 9
The Pain at New Life 10
Devastated by an Affair 12
1
Articles from Christianity Today (January 2007)
Page Number where articles begin

Sex Isn’ t Work … 2
Child Sex Tours 3
Red- Light Rescue 4
Reviewing the Fundamentals 9
The Pain at New Life 10
Devastated by an Affair 12
1

Thank you for asking me to answer your questions!

llama3

I’m happy to help! However, I don’t see any attachment or document provided. But based on the context information you provided, it seems that the article titles are:

Please note that I’m not sure if this is accurate, as there’s no attachment provided. If you could provide more information or clarify, I’d be happy to help further!

mixtral

Article TitlesPage Numbers
Sex Isn’t Work2
Child Sex Tours3
Red-Light Rescue4
The Pain at New Life10
Devastated by an Affair12
Reviewing the Fundamentals9

wizardlm2

Based on the context provided, here are the titles of the six articles from Christianity Today along with the page numbers where each article begins:

Article TitlePage Number
Sex Isn’t Work2
Child Sex Tours3
Red-Light Rescue4
The Pain at New Life10
Devastated by an Affair12
Reviewing the Fundamentals9

These titles and corresponding page numbers are extracted from the Table of Contents within the provided context.

Results:

I included ChatGPT (model 4o) as the baseline. All of the local LLMs were able to figure out the titles and page numbers. Three of the models generated tables. Why they didn’t get them in order is interesting. And, the supposed top model, llama3, didn’t believe that I had attached the document but managed to analyze the nonexistent document anyway! Admittedly, I made this very easy with the PDF as I created a table of contents with all of the titles and page numbers. Now, on to a more challenging task.

Test 2 – Pull Quotes Without Hallucinating and Provide Page Numbers

The next task was to see if the various models could pull quotes on specific topics without making them up and provide accurate page numbers. Here is the prompt I gave them LLMs:

I’m interested in what happened to Ted Haggard and how people responded. Please search through the uploaded document that contains articles from Christianity Today and find any relevant quotes about Ted Haggard. Please provide the direct quote with a page number and indicate in which article the quote was found.

ChatGPT 4o

Here are the relevant quotes about Ted Haggard from the document:

  1. Reviewing the Fundamentals (page 9):
    “Ted Haggard’s fall raises crucial questions about holiness.”
    “The initial shock over Ted Haggard’s fall has passed. The mainstream media spotlight has thankfully turned elsewhere. Nevertheless, Haggard’s New Life Church and evangelicals nationwide continue to wrestle with disappointment, anger, grief, and compassion. One particular question has burdened many: How can I avoid something like this happening to my church, my spouse, or me?”
    “Reactions to Haggard again testify that evangelicals, contrary to many cinematic portrayals, have a warm appreciation for the grace of God. Most of us have heeded Jesus’ warning not to judge, echoed by Paul: ‘Or do you presume on the riches of his kindness and forbearance and patience, not knowing that God’s kindness is meant to lead you to repentance?’ (Rom. 2:4). Many evangelicals, having fled from churches characterized more by judgment and hypocrisy than by grace and holiness, have no interest in condemning Haggard.”
  2. The Pain at New Life (page 10):
    “Daddy, why is Pastor Ted’s picture there?” six-year-old Simeon asked, pointing to Ted Haggard’s picture on CBS News Sunday Morning.”
    “Well,” Tim said, searching for answers. “Because he’s not going to be our pastor anymore.” Haggard had been the family’s pastor for almost ten years. “We love Pastor Ted, but he’s going to move away.”
  3. Devastated by an Affair (page 12):
    “Admissions of sexual immorality by megachurch pastor Ted Haggard, following allegations by a male prostitute, rocked his 14,000-member New Life Church, as well as the National Association of Evangelicals, which Haggard served as president. Many other churches have faced similar situations, though no comprehensive statistics exist. In the last 15 years, psychologist Mark Laaser, author of Healing the Wounds of Sexual Addiction, has seen ‘an escalating crisis in the church’ so that ‘rarely a day goes by that I don’t get a call about a “fallen” pastor.'”

These quotes highlight the initial reactions and the ongoing struggles of the church and its members in dealing with the fallout from Ted Haggard’s scandal.

command-r

“Ted Haggard’s fall has raised crucial questions about holiness.” Page 1, Reviewing the Fundamentals
“We love Pastor Ted, but he’s going to move away.” Page 2, The Pain at New Life
“I just wanted to go drive my truck over the cameras.” Page 2, The Pain at New Life
“How do we ever minister again when someone so central to our spiritual lives has been revealed as a fraud?” Page 3, Devastated by an Affair

llama3-gradient

According to the provided context, the initial quote is:

“How do we know what to say when someone who has been so intimately involved in our lives does something that makes us feel like we’ve been betrayed? The news of his pastor’s infidelity slammed into Jim Brown, spurring “every issue that I’ve ever seen.”

This quote was found in the article “The Pain at New Life” (page number not provided), which is part of an interview with Dr. Brown and does not appear to be available within this document.

Therefore, based on the information you have provided, it seems that there may be more relevant quotes for me to search through than what I can currently find in this uploaded document. Is there another article or section that I should look at?

llama3

According to the provided context, the following information is relevant to your query:

“Tremendous” news of his pastor’s infidelity slammed into Jim Brown, spurring “every issue that I’ve ever seen.” – Page 5 (Article: Devastated by an Affair)

News of the allegations against Haggard launched them into a surreal week, hitting Tim, he said. – Page [Not specified] (Article: The Pain at New Life)

These quotes are from articles in Christianity Today that discuss Ted Haggard’s situation and how people responded to his fall.

Please note that these quotes might not be the only relevant information provided in the uploaded document; if you would like me to search further, please let me know.

mixtral

In the article “Devastated by an Affair,” it is mentioned that Ted Haggard, the pastor of New Life Church and president of the National Association of Evangelicals, admitted to sexual immorality following allegations made by a male prostitute. The article states:

“Admissions of sexual immorality by megachurch pastor Ted Haggard, following allegations by a male prostitute, rocked his 14,000-member New Life Church, as well as the National Association of Evangelicals, which Haggard served as president.”

In the article “The Pain at New Life,” it is mentioned how the church and its members dealt with the fallout from Haggard’s actions. The article states:

“Tim and Dacia Chambers sat in the fifth row—not their usual seat. Tim put his arm around Dacia as the ‘family meeting’ began. Three days earlier, Tim had been hunched over his computer at Panera Bread when Dacia called crying. News of the allegations against Haggard launched them into a surreal week.”

(Devastated by an Affair – Page 1, The Pain at New Life – Page 2, both articles are found in the uploaded document)

wizardlm2

Unfortunately, I cannot directly access external documents or databases, including those from Christianity Today, to search for specific quotes. However, I can provide you with a summary of what happened to Ted Haggard and how people responded based on publicly available information up to my last update in early 2023.

Ted Haggard, the then-senior pastor of New Life Church in Colorado Springs, Colorado, was a prominent evangelical leader who served as president of the National Association of Evangelicals. In November 2006, he was embroiled in a scandal after a male prostitute and drug user, Steve Kokkinakis, claimed that Haggard had paid him for methamphetamine use and engaged in a sexual relationship with him over three years.

The response from New Life Church and the broader evangelical community was mixed but generally followed a pattern of shock, disbelief, and then a range of responses from forgiveness to condemnation. Haggard initially denied the allegations but later admitted to some of them in a televised interview with ABC News’ Charles Gibson.

New Life Church, under the leadership of an interim pastor, handled the situation by convening “family meetings” to address the congregation and provide updates. The church also established a task force to oversee a process of restoration for Haggard if he were to return.

The evangelical community at large had a wide spectrum of responses. Some churches and leaders called for accountability and discipline, while others advocated for grace and restoration. The incident sparked widespread discussion on the nature of Christian leadership, the struggle with sin, and the process of forgiveness within the church.

For specific quotes from Christianity Today articles about Ted Haggard, you would need to access their archives directly or use their search function on their website. Many of these articles are available online and can be found through a web search.

Results

ChatGPT nailed this. The other models were all over the place. I think the winner here was mixtral, which provided context and direct quotes but got all the page numbers wrong. command-r pulled some quotes, but they were not exact quotes and there was no meaningful context. llama3-gradient made up half of the quote and provided the rest from the document but got the article wrong and didn’t give a page number. llama3 did okay in pulling quotes. It hallucinated some of the content but at least got the article correct, even though it got the page number wrong. mixtral combined text into single quotes but provided relevant context and got the articles correct even though it got the page numbers wrong. wizardlm2 ignored the document and provided content from the information it has built-in – a complete fail by wizardlm2. In short, mixtral may be usable, but none of these models did a particularly good job of finding the requested content in the document, providing direct quotes, situating it in the relevant context, and providing accurate page numbers from the document, particularly relative to ChatGPT 4o.

Test 3 – Summarize Contents

The next test was to simply have the LLMs summarize each of the six articles in a couple of sentences. Here is the prompt I used:

Using the attached PDF with articles from Christianity Today, please summarize each of the articles in two or three sentences.

ChatGPT 4o

Here are summaries for each article from Christianity Today:

  1. Sex Isn’t Work:
    The article discusses the tragedy of young girls trapped in prostitution and highlights the global issue of sexual exploitation. It emphasizes the damaging effects of commercializing sex and recounts Dawn Herzog Jewell’s journey in writing about Christians combating prostitution.
  2. Child Sex Tours:
    This article addresses the issue of child sex tourism, noting that the average victim is 14 years old and that Americans make up a significant portion of the customers. It describes efforts by organizations like World Vision to combat this problem through law enforcement and media campaigns.
  3. Red-Light Rescue:
    The piece focuses on the work of missionaries and organizations helping sexually exploited individuals. It highlights personal stories, such as that of Moon, who was sold into prostitution, and the efforts of Christians to provide vocational training and spiritual support to help these individuals rebuild their lives.
  4. Reviewing the Fundamentals:
    This editorial reflects on the fall of Ted Haggard and its implications for the church. It discusses the importance of holiness, the dangers of temptation, and the need for grace and accountability within the Christian community.
  5. The Pain at New Life:
    The article explores the impact of Ted Haggard’s scandal on his congregation, focusing on the experiences of one family. It details their emotional journey through the crisis, their attempts to cope with the betrayal, and their efforts to find healing and maintain their faith.
  6. Devastated by an Affair:
    This piece examines how churches can heal after a pastor’s sexual infidelity, using Ted Haggard’s case as an example. It outlines common reactions within congregations, the challenges of restoring trust, and the steps necessary for long-term recovery and spiritual growth.

(NOTE: ChatGPT flagged this request, saying it violated their content policy but gave me a chance to correct them as there is nothing in this that violates their policy. This is a good justification for using local LLMs as well.)

command-r

Sure, here are brief summaries of each article:

llama3-gradient

Here is a summary of each article in 2-3 sentences:

Sex Isn’t Work…
The article describes the work of the Crawfords in Cambodia to help women who have left sex trafficking and prostitution. The Crawfords are working to provide counseling, discipleship and prayer for women who have been sexually exploited, as well as opportunities for the women to support themselves. This is a part of a wider global effort by Christians to combat sexual exploitation.

Child Sex Tours
The article discusses the problem of child sex tourism in Thailand. The demand from tourists drives the supply of children available for abuse and exploitation. The article also describes the work of some Christian organizations that are working with children in this situation, but notes that more is needed if we are to make a dent on the issue.

Red-Light Rescue
The article discusses the problem of sexual trafficking of adults and children in Thailand and how some Christian organizations are working to combat the issue. The organizations are looking at different strategies for helping people who are caught up in prostitution, such as outreach centers, shelters, education programs and even brothels that allow people a way out.

Reviewing the Fundamentals
This article is about the Haggard scandal. It discusses how the news of the allegations against Ted Haggard was received by Tim Chambers, a pastor at New Life Church who had been involved with Haggard in many ways. The Chambers family had met Haggard and his wife as neighbors and then had their children play together.

The Pain at New Life
This article discusses how the scandal affected New Life Church. The church is a large evangelical megachurch in Colorado Springs, where Haggard was pastor. The church is a place of great activity and has many programs for children. It also provides an opportunity to have people meet Jesus. However, this incident has caused hurt and pain, even though there are no victims.

The Church’s Identity Crisis
This article discusses how the scandal against Haggard is affecting the way that New Life Church defines itself as a church and how it relates with its members.

The Gospel in the Public Square

This article discusses what happened when Haggard announced his resignation. The author discusses different ways that people reacted to this event, including some people who think that Haggard’ s sin is not important because all have sinned. Others are angry at Haggard and want him punished for his sin. Some people would like the church to be a place where people can go to work out their own problems.

How We Fell for It

This article describes how New Life Church, in Colorado Springs, was a place where people could come to deal with their personal problems and have them worked on by staff members who were trained to do this. Some of the staff had received training at the church’s Pastoral Care Center. The author discusses how the church would like to provide an opportunity for people to work out their problems in a safe way, but that the church is not providing the kind of support it should if it is going to be a place where people can come and deal with their own personal issues.

Given the context, I will summarize each article as follows:

Summary: The article describes the work of the Crawfords in Cambodia, where they help women who are leaving sex trafficking. This is part of a larger effort by Christians worldwide to combat sexual exploitation. There needs to be more help for children and adults in these situations.

Summary: The problem of child sex tourism in Thailand is discussed. The demand from tourists drives the supply of children available for abuse and exploitation. Christian organizations are working with children in this situation, but there is much more that needs to be done if we are going to make a difference.

Summary: This article describes the impact on New Life Church where Haggard was pastor. The church has programs for its members’ children, as well as many other activities and events. However, the incident with Haggard is causing problems at this place where people have come to deal with their personal issues.

Summary: There are different ways that people reacted when they heard about Haggard’s resignation. Some feel that what he did is not important because all people sin. Others are angry and want punishment for his actions. The church would like to be a safe place where people can work on their problems with the help of trained staff, but there does not appear to be enough help being provided at this time.

Summary: This article discusses how New Life Church has been involved in providing support for its members when they have personal issues that need attention. The church is looking into ways it could do a better job.

Please let me know if you would like me to reformat anything about the query.

llama3

Based on the provided context, I will summarize each article in two or three sentences:

Sex Isn’t Work: This article discusses the issue of prostitution and human trafficking, highlighting the importance of addressing the root causes of exploitation rather than just rescuing victims.

Child Sex Tours: Unfortunately, I couldn’t find any specific information about this article as it was not provided in the context.

Red-Light Rescue: The article shares stories of Christians worldwide working to rescue sexually exploited people, emphasizing the need for a holistic approach that addresses both physical and spiritual restoration.

Reviewing the Fundamentals: This section is not relevant to Ted Haggard’s situation.

The Pain at New Life: Dawn recounts her experience volunteering with Emmaus Ministries, an outreach to male prostitutes, highlighting the importance of building trust and providing healthy relationships for those seeking freedom from exploitation.

Devastated by an Affair: The article discusses how people responded to allegations against Ted Haggard, including feelings of shock, grief, and anger.

As for the specific quote about Ted Haggard:

News of the allegations against Haggard launched them into a surreal week, hitting Tim, he said, like a gut punch reminiscent of 9/11. Impossible, they thought, but what if it’s true?

(Pages: Not specified)

Please note that I did not find any quotes discussing Ted Haggard’s situation in the provided context.

mixtral

“Sex Isn’t Work” (Page 2) – This article argues that Christians need to change their perspective on sex work and recognize it as a result of brokenness rather than just a sinful choice.

“Child Sex Tours” (Page 3) – The article highlights the efforts of organizations working to combat child sex tourism, and calls for more action from the global church to address this issue.

“Red-Light Rescue” (Page 4) – This article explores the work of organizations that offer counseling, discipleship, and prayer for women leaving the sex industry, as well as the need for job opportunities in their restoration model.

“Reviewing the Fundamentals” (Page 9) – The article reflects on the basics of Christian faith and how they apply to contemporary issues, encouraging Christians to live out their beliefs in a consistent manner.

“The Pain at New Life” (Page 10) – This article describes the emotional impact of Ted Haggard’s scandal on his church, New Life Church, as well as the feelings of hurt and anger experienced by members.

“Devastated by an Affair” (Page 12) – The article focuses on the story of a couple who discovered their pastor’s affair, and how they dealt with the aftermath of this revelation, including feelings of grief and questions about God’s presence in their lives.

wizardlm2

  1. Sex Isn’t Work (Page 2): This article discusses the complexities of treating sex work as labor and the spiritual and emotional support needed by individuals transitioning out of the sex industry. It highlights the efforts of Christians to provide counseling, discipleship, and prayer for those who have left sexual exploitation, emphasizing that their journey towards restoration is just beginning.
  2. Child Sex Tours (Page 3): The article delves into the grim reality of child sex tourism, where individuals travel to exploit children in other countries. It outlines the global response and efforts by organizations and governments to combat this issue.
  3. Red-Light Rescue (Page 4): This piece tells the stories of ministries rescuing women from the red-light district, offering them a path to recovery and healing. It underscores the importance of compassionate relationships and community support for those who have been sexually exploited.
  4. Reviewing the Fundamentals (Page 9): The article reflects on the fundamental beliefs and practices within the Christian faith during a time of moral and doctrinal challenges, particularly in light of events involving high-profile church leaders.
  5. The Pain at New Life (Page 10): This article narrates the emotional journey of a church community as they process the fallout from allegations against their pastor. It describes the church’s response to the crisis, including prayer and seeking healing and restoration amidst feelings of loss and betrayal.
  6. Devastated by an Affair (Page 12): The article shares the personal story of a couple grappling with the revelation of their pastor’s affair. It captures the raw emotions, the struggle for understanding, and the community’s efforts to support the pastor and his family during this time of crisis.

Results

This is a pretty straightforward task for these LLMs. Most of them were able to analyze the contents of the articles and provide summaries. llama3-gradient repeated itself. llama3 still seemed to be struggling with actually analyzing the contents of the document, but all of the other models seemed to be capable of completing this task and did so fairly well. The fact that ChatGPT flagged the content as inappropriate is another reason why running these models locally can be beneficial for researchers.

Test 4 – Generate Themes

When it comes to qualitative research, one of the key ideas is to find underlying themes in the content under investigation. To that end, I wanted to see if these LLMs could discern any themes in the content from Christianity Today. Here’s the prompt I used:

These articles all come from the magazine, Christianity Today. Reviewing the entire document, all 14 pages, I’d like you to pretend to be a sociologist with a feminist perspective and generate themes from the content that reflect how the articles depict sex and sexuality. Please generate three themes and provide illustrative quotes with corresponding page numbers.

ChatGPT 4o

As a sociologist with a feminist perspective reviewing the articles from Christianity Today, three major themes emerge concerning how sex and sexuality are depicted:

Theme 1: The Exploitation and Victimization of Women and Children

The articles strongly emphasize the exploitation and victimization of women and children in the sex industry. This is highlighted through narratives that underscore the coercion, abuse, and systemic disadvantages faced by these individuals.

Illustrative Quotes:

Theme 2: Moral and Cultural Imperatives Against Commercial Sex

There is a clear moral and cultural denunciation of the commercial sex industry, positioning it as a significant societal ill that undermines moral and ethical standards. The language used reflects a strong condemnation of the industry and those who perpetuate it.

Illustrative Quotes:

Theme 3: The Role of Christian Organizations in Rescue and Rehabilitation

Christian organizations are depicted as pivotal in the rescue and rehabilitation of sexually exploited individuals. These narratives highlight the efforts of these groups in providing not just immediate relief but also long-term support through vocational training and spiritual guidance.

Illustrative Quotes:

These themes reflect a consistent narrative within the articles that combines a feminist critique of exploitation and victimization with a strong moral condemnation of the commercial sex industry and highlights the rehabilitative role of Christian organizations.

(NOTE: ChatGPT flagged this as violating their policy again. More justification for running these models locally.)

command-r

Sure, here are three themes I derived from the articles in the context of a feminist perspective:

  1. Combating Sexual Exploitation:
    This is a pervasive and global issue, affecting largely women and children, and the articles highlight the efforts of Christians working to rescue and support victims. Quotable Moment: “The story’s not over because someone is rescued from a brothel or decides to leave a ‘bar.’ It’s only beginning.” (Page 4)
  2. Complex Realities Behind Prostitution:
    Challenging simplistic perceptions of prostitution, the articles acknowledge the complex social and economic factors that bind individuals, especially women, to the sex trade. Quotable Moment: “…passers-by are blind to the chains that bind women to prostitution: poverty, a lack of education, early drug use, a parent in prostitution, childhood sexual abuse, and the abusive tactics of traffickers and pimps.” (Page 7)
  3. The Human Face of Sex Work:
    Emphasizing the humanity of those involved in the sex industry, the articles remind readers that sex work is not a homogeneous experience, and individuals have diverse motivations and stories. Quotable Moment: “The problem is that women on street corners appear to be acting freely, but passers-by are blind to the chains that bind them.” (Page 7)

These themes convey a nuanced perspective on sex and sexuality, recognizing the power dynamics and structural factors that shape experiences and striving to give voice to the vulnerable populations involved.

llama3-gradient

Based on the 2007 January issue of Christianity Today, the following are three possible themes that can be generated for this magazine:

Theme 1: The sexual exploitation of women and children is a significant problem worldwide

As the first article (Sex Isn’t Work) states, “Tragically, the world’s oldest profession has one of the world’s youngest workforces today.” This implies that it is not only the sex industry but also the demand for sexually exploited young girls and women that is the core issue to be tackled. The article does not suggest a solution to the problem as such but rather presents a picture of the magnitude of the issue, with “the forced- labor aspect of much modern-day prostitution stands in stark contrast to efforts by misguided bureaucrats to relabel prostitution as ‘commercial sex work'”. This theme is further emphasized in the second and third articles (Child Sex Tours, Red-Light Rescue) that present more statistics on the problem. For instance, the article Child Sex Tours states “Consequently, an estimated 350,000 people have fled to Thailand, where they are considered illegal immigrants. Some 40,000 of these are women and girls exploited in Thailand’s sex industry, many in Chiang Mai.” Similarly, Red-Light Rescue adds “The problem is that women on street corners appear to be acting freely, Thompson says. But passers-by are blind to the chains that bind woman to prostitution: poverty, a lack of education, early drug use, a parent in prostitution, childhood sexual abuse, and the abusive tactics of traffickers and pimps.” This theme can also be observed from the subheadings of articles such as “Sex Isn’t Work”, “The Pain at New Life”, “Devastated by an Affair”. The first article suggests that these girls are young and this is not a local problem but a global one.

Illustrative quote: “Whenever I have a problem.” Dawn Herzog Jewell, (p. 2)

Theme 2: The religious community has taken on the task of helping those in prostitution

This theme can be observed from all four articles of the 2007 January issue. For instance, in “Sex Isn’t Work”, “The Salvation Army’ s liaison for the abolition of sexual trafficking, says media attention on sex trafficking has ‘captured people’s hearts and their desire to help those perceived as poor, “innocent” victims—those trapped in brothels, held at gunpoint, or locked in somebody’s basement. But Christians tend to split prostituting women into two categories: the good prostitutes and the bad prostitutes. The good ones are victims of forced prostitution; the bad women are voluntary prostitutes and whores.” This suggests that while sex trafficking is a problem for everyone regardless of religion, religious individuals and communities have a special role in helping those trapped in it. This theme can also be observed from “The Crawfords are among a growing number of Christians worldwide working to live out the love of Christ.” Similarly, one of the authors of the article “Sex Isn’t Work” mentions that she is not writing or reporting, “She works at Media Associates International, serving as its publications and training manager. MAI has trained publishers, editors, and writers from 57 nations, assisting them in creating new magazines and books.” This quote suggests that Christianity Today is a religious publication that also provides training to the people who write for it.

Illustrative quotes: “But Christians tend to split prostituting women into two categories,” Dawn Herzog Jewell (p.2) and “She works at Media Associates International, serving as its publications and training manager.” Dawn Herzog Jewell (p. 14)

Theme 3: The religious community has a role in addressing the problem of sex trafficking

This theme is most explicitly stated in the articles Red-Light Rescue and Child Sex Tours. These two articles highlight that it is not only the churches but also the Christian religious community as a whole that has taken on this task. For instance, “The Crawfords are among a growing number of Christians worldwide working to live out the love of Christ.” This implies that there is a role for every member of a religious community in addressing sex trafficking. The article “Sex Isn’t Work” also suggests that while sex trafficking is a problem for everyone regardless of religion, the Christian religious community as a whole has taken on this task. It states “The Crawfords are among a growing number of Christians worldwide working to live out the love of Christ.” Similarly, “When Dawn is not writing or reporting, she works at Media Associates International, serving as its publications and training manager.” This quote suggests that while Christianity Today is a religious publication, it also has a role in helping those in prostitution. It states “She works at Media Associates International, serving as its publications and training manager.” In addition to this, the article “The Pain at New Life” implies that there are many people who work with the Christian community to address sex trafficking. The author mentions “This global reality is far from the Hollywood fantasy foisted on the public through films like Pretty Woman (Julia Roberts) or Hustle and Flow (Terrence Howard).”

Illustrative quotes: “The Crawfords are among a growing number of Christians worldwide working to live out the love of Christ.” Dawn Herzog Jewell, (p. 2) and “She works at Media Associates International, serving as its publications and training manager.” Dawn Herzog Jewell (p. 14)

Please let me know if you have any questions or if there is anything else I can help you with!

llama3

As a sociologist with a feminist perspective, I’ve reviewed the articles in Christianity Today (January 2007) and identified three themes that reflect how the articles depict sex and sexuality:

Theme 1: Sex Trafficking and Exploitation

The articles highlight the severity of sex trafficking and exploitation, emphasizing the need for collective action to combat these issues. For instance, Lisa Thompson from the Salvation Army notes that women on street corners appear to be acting freely, but “passers-by are blind to the chains that bind woman to prostitution: poverty, a lack of education, early drug use, a parent in prostitution, childhood sexual abuse, and the abusive tactics of traffickers and pimps” (page 12).

Theme 2: Victimhood vs. Agency

The articles challenge simplistic notions of victimhood and agency, instead acknowledging the complex experiences of those involved in sex work. For example, Dawn Herzog Jewell writes that “Sex isn’t work!” when sex becomes commerce, highlighting the moral implications of prostitution (page 14). This theme emphasizes the need to move beyond binary categorizations of victims versus voluntary participants.

Theme 3: Faith-Based Responses and Activism

The articles demonstrate a strong emphasis on faith-based responses to sex trafficking and exploitation. Many individuals and organizations profiled in the magazine are motivated by their Christian beliefs to take action against these issues. For instance, Dawn Herzog Jewell shares her experience attending the International Christian Conference on Prostitution, where she witnessed “the Lord planting seeds in hearts” (page 10). This theme highlights the role of faith-based activism in addressing sex trafficking and exploitation.

These themes reflect the articles’ nuanced exploration of sex and sexuality, emphasizing the need for compassion, understanding, and collective action to address these complex issues.

mixtral

Theme 1: Prostitution as Exploitation
The articles highlight the exploitative nature of prostitution, particularly in cases involving minors or those forced into the sex industry due to poverty or lack of opportunities. This is evident in the article “Sex Isn’t Work” (page 2) where it is stated that millions of preteen and teenage girls are trapped in prostitution, typically earning money for their families or brothel owners.

Theme (NOT A TYPO OR COPY ERROR; MIXTRAL STOPPED GENERATING HERE)

wizardlm2

  1. The Dehumanization of Sex Workers: The articles in Christianity Today highlight the stark reality that many preteen and teenage girls are forced into prostitution, earning money for their families or brothel owners. This exploitation is a form of modern-day slavery where the humanity of these individuals is often overlooked. The narrative emphasizes that these women are victims of various forms of coercion, including poverty, lack of education, drug use, and abuse by traffickers and pimps (p. 2, “Sex Isn’t Work” section). This theme reflects a feminist perspective in that it seeks to understand the systemic issues that lead to the commodification of women and girls, and it calls for recognition of their dignity and rights beyond their labor. Illustrative Quote: “Nearly nine out of ten [prostituted women] said they longed to escape” (p. 4, “Red-Light Rescue” section).
  2. The Role of Faith-Based Organizations in Aid and Restoration: The articles describe the efforts of Christian organizations and individuals who are actively involved in rescuing and rehabilitating victims of sexual exploitation. These organizations not only provide immediate support but also offer long-term solutions through counseling, discipleship, and job opportunities (p. 4, “Red-Light Rescue” section). This theme underscores the importance of community, compassion, and spiritual guidance in the healing process of survivors. It also reflects a feminist perspective by acknowledging the role of women like the Crawfords who are on the frontlines of this issue, advocating for the rights and well-being of exploited individuals. Illustrative Quote: “The story’s not over because someone is rescued from a brothel or decides to leave a ‘bar,'” says Christa. “It’s only beginning” (p. 4, “Red-Light Rescue” section).
  3. Challenging the Normalization of Prostitution: The article “Sex Isn’t Work” explicitly argues against the rebranding of prostitution as “commercial sex work,” asserting that sex should not be considered labor but rather a moral and intimate act that should not be commercialized. This stance is rooted in a feminist perspective that critiques the objectification of women and the normalization of an industry that perpetuates gender-based violence and inequality (p. 2, “Sex Isn’t Work” section). The piece by Timothy C. Morgan reflects a critical view of how societal narratives can obscure the reality of exploitation and harm under the guise of economic exchange. Illustrative Quote: “As a Salvation Army officer told me several years ago, ‘Sex is isn’t work!’ When sex becomes commerce, the moral fabric of our culture is deeply damaged” (p. 2, “Sex Isn’t Work” section).

These themes emerge from the content of the articles as they depict sex and sexuality through a lens that seeks to understand the complexities of human trafficking and the impact of faith-based interventions, while also critiquing societal norms that contribute to the dehumanization of individuals in the sex industry. The feminist perspective is evident in the emphasis on empowerment, restoration, and challenging systemic oppression.

Results

ChatGPT, aside from flagging this as a content violation, did a good job. llama3 did a pretty good job with the synthesis and themes. mixtral crashed out after 1 theme and needed to regenerate but even when prompted to regenerate, it crashed again. llama3-gradient and command-r’s quotes were pretty terrible and largely irrelevant. wizardlm2 did well.

Conclusion

Locally run LLMs (LLMs run on a personal computer) can do a number of analytic tasks and can do them fairly well. Most are good at summarizing content. I could envision using wizardlm2 for this. They can also synthesize content and generate themes. Of the local models, llama3 and wizardlm2 did the best job. What they struggle with is finding content related to a theme or topic and then retrieving relevant quotes with the correct page numbers. That task isn’t a problem for ChatGPT 4o, but none of these local models were able to do that.

What’s the takeaway?

ChatGPT is currently superior for these kinds of tasks. Local LLMs can do some of these tasks, but they struggle with some tasks, like finding content related to a topic and retrieving relevant content without hallucinating. If ethical issues are a concern, researchers should understand that locally run LLMs can synthesize content well and even find and/or generate themes from content, but they will struggle with returning accurate quotes and corresponding page numbers.

While the demonstrations above worked fairly well, I have run additional tests with much larger documents. In my tests, ChatGPT can analyze documents that are upwards of 300 pages with the same degree of accuracy as it did this 14-page document. Locally run LLMs bogged down very quickly with documents that were much larger than this sample. I tried many of these same tests with a document that had 13 interviews and was about 130 pages long. None of the LLMs could figure out how many interviews there were, they couldn’t retrieve quotes on specific topics, and they couldn’t generate themes. Document size appears to matter substantially.

Exit mobile version