Evaluating Web of Science’s New Search Engine

How I tested, evaluated, and influenced the rollout of One-Box Smart Search across one of the most trusted platforms in academic research.

Reimagining the Web of Science Search Experience

Client

Web of Science (part of Clarivate)

Role

Solo UX Researcher

Stakeholders

PMs, Search Engineering team, UX team

Duration

Feb 10th - March 6th 2025

Presented On

March 14th 2025

RESEARCH READOUT PRESENTED TO STAKEHOLDERS

Final Deck

LONG-TERM INFLUENCE

Outcome & Impact

The research insights directly informed key product decisions ahead of Smart Search’s global release used by over 20 million researchers across 4,700+ institutions.

Smart Search launched in April 2025 as an opt-in release. Following this study, the July 2025 release shipped to all users by default with several updates that reflected this study's research findings.

Research Insight

Recommendation

What Was Implemented, July 2025 Release

01
Boolean logic was misunderstood by advanced users. They didn’t know if the system honored their syntax.

Provide feedback cues to show how Boolean queries are interpreted.

✅ Smart Search now clarifies how logic is applied, improving transparency and trust.

02
Labels and terms were ambiguous. Users didn’t understand terms like Semantic Result, Editor Records, or Citation Count.

Add tooltips and clearer terminology for technical or uncommon labels.

✅ Tooltips were added to explain ambiguous terms like “Semantic Search” and “Boolean Search”

03
Users were uncertain about researcher ranking.

Enhance transparency by providing insights into how search results are

ranked and matched to user queries.

Enhance transparency by providing insights into how search results are

ranked and matched to user queries.

Enhance transparency by providing insights into how search results are

ranked and matched to user queries.

✅ Researcher tab now includes clearer metadata and logic explanations for sorting.

04
Advanced researchers (e.g., professors, PhDs) preferred using Boolean operators and the traditional Advanced Search interface.

Allow expert users to continue using Advanced Search alongside Smart Search.

✅ Advanced Search remains available as an opt-out option to respect habits of power users.

05
Users didn’t understand what “Semantic Search” and "Boolean Search" meant

Help users understand how their queries are processed without needing to decode the system.

✅ The update added tooltips and labels to explain Boolean, Semantic, and Combined logic

MY ROLE

Led End-to-End Research

01 SUMMARY

What is Web of Science?

Web of Science is a leading academic search platform used by researchers to find trusted, peer-reviewed publications. It helps users explore scholarly articles, track citations, and navigate the connections between research across disciplines. Known for its quality and precision, it’s a core tool in the academic research workflow.

02 SUMMARY

At a Glance

20M+ researchers rely on it globally
4,700+ institutions subscribe across 98 countries
250+ disciplines covered, with a focus on high-impact, peer-reviewed sources
Over 240M indexed records including journals, books, and proceedings
2.1B+ cited references and 1.9M+ open access articles

03 SUMMARY

Project in a Nutshell

This was a time-sensitive research study focused on evaluating the early version of “Smart Search” on Web of Science—a core search engine used by over 10 million researchers. While Web of Science is widely trusted for its curated, peer-reviewed academic content, its Advanced Search feature was often too complex for everyday use. To address this, the team developed a simplified, AI-supported one-box search interface called Smart Search.

I led the UX research for this project from designing the study to recruiting users, conducting interviews, analyzing findings, and presenting results to Clarivate stakeholders. My work directly shaped product decisions on logic handling, UI design, and search transparency.

LEGACY SEARCH TOOL TRUSTED BY MILLIONS OF RESEARCHERS

Advanced Search

Advanced Search on Web of Science is a Boolean-based interface built for expert users. It lets researchers create precise queries using field tags (like `AU=` for author), logic operators (`AND`, `OR`, `NOT`), and custom filters. This level of control is great for deep, systematic searches, but it requires knowing the right syntax and structure.

Smart Search

Smart Search is a new addition to Web of Science, a simplified, AI-enhanced search bar designed to feel more like Google. Users can type natural language queries without needing field codes or Boolean logic.

The feature was ready, but was it right?

In January 2025, the product was already being built, but the team needed a pulse check:

Would this new Smart Search experience actually meet the needs of real researchers?

My research aimed to answer these questions by validating usability, transparency, and search relevance before launch.

KEY AREAS OF INQUIRY

Research Questions

What do researchers expect from Smart Search, and how do they see it fitting into their existing workflows?

How intuitive is Smart Search for locating research materials?

What pain points, if any, do users encounter while using Smart Search?

How do search suggestions (typeaheads) and smart filters affect researchers’ ability to complete tasks?

How relevant and complete do users perceive Smart Search results to be?

How does user satisfaction compare between Smart Search and Advanced Search?

01 THE PROCESS

Participant Recruitment

We recruited 8 participants representing diverse academic levels and research backgrounds. All were verified users of Web of Science with recent research activity.

02 THE PROCESS

Disciplines Covered

We ensured that our participants spanned beginner to advanced research skill levels, from exploratory undergrads to methodical professors. This diversity helped us see differences in mental models, expectations, and frustrations.

Biomedical Science

Chemical Engineering

Pyschology

Business Law

Literature

Information Science

03 THE PROCESS

Recruitment Platform

04 THE PROCESS

Methodology

Sessions were remote, moderated usability studies lasting ~60 minutes. The protocol was designed to mirror a real academic search workflow, using participant-led search queries in their own fields.

The Results

WHAT WE HEARD FROM USERS: KEY INSIGHT 01

"I trust Web of Science because I know the content is good."

Credibility is the core reason people use Web of Science. They need to know they’re seeing peer-reviewed, complete, and high-quality results. Any new feature especially something driven by AI must continue to uphold that trust.

WHAT WE HEARD FROM USERS: KEY INSIGHT 02

"I want to know how it’s choosing these results."

Users were curious (and cautious) about how search suggestions and results were being generated. They didn’t care whether it was Boolean or AI, they just wanted it to be accurate and to understand the system’s logic.

WHAT WE HEARD FROM USERS: KEY INSIGHT 03

"Some key papers are missing, I know Google Scholar has more results"

When expected papers didn’t show up, trust dropped immediately. Researchers often already know what kind of publications should appear for their queries. If they’re missing, they look elsewhere.

WHAT WE HEARD FROM USERS: KEY INSIGHT 04

"I typed a Boolean string but it didn’t behave like one."

Advanced users brought their Boolean habits into Smart Search. But because Smart Search converts queries in the background, users got confused when their usual logic didn’t produce familiar results.

WHAT WE HEARD FROM USERS: KEY INSIGHT 05

"I always double check with other databases."

These users aren’t casual searchers, they’re methodical. They cross-check results in Google Scholar, ProQuest, and PubMed. If WoS doesn’t show a full picture, they move on.

QUANTITATIVE FEEDBACK

Comparing the Two Interfaces

To understand how researchers perceived each tool, we asked participants to rate both Smart Search and Advanced Search across five dimensions: relevance, accuracy, trustworthiness, quality, and overall helpfulness.

Smart Search consistently scored higher on relevance, accuracy, and perceived helpfulness, reflecting its ease of use and intuitive experience.

While Smart Search was easier to use, some participants still perceived Advanced Search as delivering higher-quality results, likely because of its structured inputs and predictable behavior.

Reflections

This was also my first time conducting research and synthesizing insights around a large search engine product used by millions of researchers, and this made the impact huge. After a senior UX researcher left in Feb, I took this on as a solo researcher. What made this project meaningful was knowing that my insights were shaping how a search engine thinks, ranks, and earns user trust.

This study challenged me to balance the needs of novice and expert users, while addressing both surface-level usability issues and deeper systemic concerns like search trust and epistemic transparency. Academic researchers have distinct cognitive and emotional demands from a search tool as they’re not casual browsers. They look for tools that not only give them results, but give them confidence in those results.

Thank you for reading!

INSIGHTS SUMMARY

Key Themes

01
Mental models mismatches around the role of NLP vs. Boolean search logic

02
Users found it difficult to interpret search result relevance without visibility into the system's reasoning

03
Users' gaps in perceived trust, especially when search results felt incomplete

04
Unclear affordances in UI text and search behavior labels

05
Frustrations around irrelevant or overloaded results