From Text to Vectors: How Embeddings Make Messy Data Useful

Three real-world patterns for turning unstructured text into usable data

At Lander Analytics, we recently worked on three projects that looked unrelated. A healthcare company needed to match its internal list of insurance plans against a large external file where the names rarely lined up. An oil and gas operator had free-form safety reports that were difficult to categorize quickly enough to be useful. Another company had roughly 3,000 open-ended survey responses sitting in a spreadsheet, which is a familiar way for valuable data to become mostly decorative.

All three projects ended up using embeddings. Embeddings gave us a common representation for the messy text. After that, the projects diverged pretty quickly. A bad insurance match, a missed safety category and an irrelevant survey result create very different problems.

Most of the work came after the vectors were generated: deciding what to compare, testing whether the results held up and figuring out which cases still needed someone to take a look.

What an embedding is

The plainest explanation I know is that an embedding turns a piece of text into a list of numbers that represents roughly what the text means, rather than only which letters it contains. Picture giving every phrase an address on a map. Phrases with similar meanings land in the same neighborhood, while unrelated phrases end up farther apart. “Cat” and “pet” should be neighbors. “Cat” and “car” should not be, even though they differ by one letter.

An embedding model might turn a short phrase into a vector containing hundreds of numbers. You will never read those numbers individually, nor would you learn much by trying. No single dimension has a clean label such as “animalness” or “insurance-plan-ness.” The vector as a whole places the text somewhere in a mathematical space.

Once the text is represented this way, we can measure how close two vectors are. Cosine similarity, which compares the angle between vectors, is a common method, although it is not the only one and the right distance measure depends on the model and the task. Modern embedding models learn these representations from large collections of text and examples of what should or should not be considered similar. That is how “physician” and “doctor” can end up close together despite sharing no overlapping letters.

The resulting similarity score gives you a place to start. You still have to decide what happens automatically, which cases need review and what the system should do when the answer is unclear.

Three patterns we keep seeing

Matching: reconciling insurance plans

The healthcare company needed to line up its internal insurance plans with a large external dataset. The naming was chaotic. For example, one plan might appear as “BlueX Cross,” “Bl Cross” and “BX CrossPlan,” with no reliable identifier shared between the two files. A normal database join found very little, while fuzzy string matching caught some obvious spelling variations but struggled with abbreviations and incomplete names.

“LA,” for example, might refer to Los Angeles or Louisiana. An embedding could surface the likely candidates. Resolving “LA” still required geography and other plan details because the name alone did not contain enough information.

Then came the less glamorous pieces. We needed candidate generation so we were not comparing every record with every other record. We needed structured fields, including geography and other plan information, to resolve cases where the name alone was not enough. We also needed thresholds that determined which matches could proceed automatically and which ones required review.

Those thresholds cannot be chosen because 0.87 feels pretty good. They need to be tested against known matches and calibrated around the cost of an error. A false positive could connect a person to the wrong plan, so the borderline cases belonged in a review queue. The embedding cut down the search space. The thresholds, structured fields and review queue kept a plausible candidate from becoming the wrong match.

Classification: sorting safety reports

The oil and gas operator wanted incident reports assigned to categories it already used, including fall hazards, equipment malfunctions and fire risks.Their existing supervised model already used structured information from each incident, along with features extracted from the written report, such as keywords. We replaced that limited representation of the free-form narrative with embeddings, using the embedding vector as an additional feature alongside the structured data in the model. 

The company’s categories came from its own safety program, which mattered because a general embedding model does not know how a particular operator defines a fire risk or which incidents demand immediate attention. The labeled examples and company taxonomy supplied that context.

Evaluation also had to happen at the category level. A model can look good on average while doing a poor job on a rare category that happens to carry the most risk. Overall accuracy can easily hide that problem. Adding the embeddings gave the classifier better information to work with. The company’s taxonomy and labeled reports were what made the predictions relevant to the safety team.

Retrieval and analysis: reading survey responses

The survey project is probably the most familiar because almost every company has a spreadsheet like this somewhere. There were roughly 3,000 open-ended responses. The conventional approach was for an analyst to spend weeks reading them, tagging themes by hand and writing a summary. That work is slow, somewhat subjective and difficult to compare with the next quarter, so the free-text responses often go unread.

We embedded each response and stored the text alongside its vector in a vector database. LanceDB is one option for this kind of setup. That allowed semantic search rather than only keyword search. Someone could search for “frustration with onboarding” and retrieve relevant responses even when respondents used phrases such as “setup was confusing” or “I could not figure out where to begin.”

New responses could be embedded as they arrived and searched alongside the older ones. We then used generative AI to extract themes, sentiment and key phrases, and structured those results into variables that could be compared across segments and tracked over time. That made it possible to watch for changes such as a rise in complaints about shipping delays without starting the analysis from scratch each quarter.

The annoying part was keeping those variables stable. A model might split one theme into two during the next run, combine two earlier themes or give an old idea a new name. For exploratory analysis, that drift may not matter much. Quarter-over-quarter reporting is less forgiving. Maintaining that comparison requires consistent categories, reproducible processing and a record of what changed. Otherwise, rerunning the pipeline can produce a different story without giving anyone a good way to explain why.

It is easy to build a demo that retrieves a few convincing responses. It is harder to make the analysis comparable six months later, after the data and the questions have changed.

Knowing when to leave embeddings alone

Not every collection of messy text needs embeddings. Some matching problems are best handled with normalization and deterministic rules. Some search problems only need full-text indexing, and a standard classifier may handle a categorization task perfectly well without adding vectors.

Embeddings are flexible, which makes them easy to reach for too soon. They also add another model, another evaluation problem and often another piece of infrastructure to maintain. We usually want a simpler baseline first: try the database join, normalized strings and fuzzy matching, full-text search or a conventional classifier. Then test whether embeddings improve the result enough to justify the additional system.

Sometimes the improvement is obvious. Other times you have built a more expensive version of a system that already worked. That is much better to learn in a small test than after the architecture is in place.

Start with the decision

Before choosing an embedding model or a vector database, get clear on what the text needs to do. Matching records, assigning categories and retrieving relevant responses are different jobs, and each one fails differently.

That should determine how the system is evaluated and where review is needed. Matching thresholds should reflect the cost of a false match. Classification errors should be examined by category, especially when rare cases carry more risk. Retrieval comes down to whether people can find what they need without digging through irrelevant results. Getting that decision right usually matters more than which embedding model you choose.

Jared P. Lander

Founder and Chief Data Scientist

Lander Analytics

---

Subscribe to our Substack and below to our monthly emails for practical AI strategies for your organization: what to build, what to avoid, and how to make systems reliable in the real world.

Work with us: If you want help identifying the right first workflow, building a permissioned knowledge base, or training your team to ship responsibly, reach out at info@landeranalytics.com.

About the author: Jared P. Lander is Chief Data Scientist and founder of Lander Analytics, where he helps organizations build practical, measurable AI workflows grounded in strong data foundations.