I led a team of rotating engineers and multiple PMs over my three years at Labelbox to iterate on and adapt our text editor to the rapidly changing needs of our customers - mostly shaped by the evolution of text labeling from simple NLP and OCR use cases, to more advanced chatbots, and finally full blown LLMs.
Introduction to the text editor
Initially, the text editor was a way for ML engineers and AI labelers to label text in documents - ultimately, the goal being to train models to recognize either individual characters for OCR use cases, or certain words or phrases of interest for NLP use cases (also known Natural Language Processing).
The v0 of the text editor therefore only supported text entities - labeling schema that could be applied to any string of text as markers for what the text should be identified as (ie "Person", "Place", "Verb", "Noun", "Car", etc.)
Relationships
Early on into my time working on the text editor, our Product team started receiving multiple requests from different companies to support a concept called "relationships". The main problem our customers were trying to solve was to have a way to draw relationships between their NER text annotations. The ultimate question that I had though, was why?After discussing with customers, two main use cases emerged:
Use case 1 (Dependency parsing): Dependency parsing refers to the process of examining the dependencies between the phrases of a sentence in order to determine its grammatical structure. Customers trying to use relationships in this way were ultimately trying to label in this manner so that they could train chatbots to speak in a more natural, human way by teaching them proper sentence and grammar structure. As a result, it is also worth noting that most of the text examples for this use case tended to be shorter (one to two lines).
Use case 2 (Coreference): Coreference NLP models produce a mapping of all the expressions in a text that refer to the same real-world entity. In short, coreference revolves around the fact that two or more expressions in a text – like pronouns or nouns – link to the same person or thing (ex. "John" and "he"). They also wanted to be able to predefine these relationships in some cases where there was a literal relationship between the two entities (ie "brother", "sister", etc.). This process, like the first use case, is fundamental to creating conversational AI. As a result of this, and in contrast to dependency parsing, most of the text examples for this use case tended to be much longer, and I learned that some of these relationships could span multiple pages.
User interviews
The main problem I was trying to solve with this work was rectifying the way users wanted to create relationships between these two use cases, and therefore how they wanted to set them up at the project level. Instead of making any assumptions, I decided to show users a prepopulated text editor screen and ask them, "How would you expect to create a relationships between two annotations?"
“I don't understand why I don't have a specific relationship tool in my ontology"
As you can see, it became immediately clear that users first expected a dedicated tool in the editor, similar to their NER annotation tools. I had the validation I was looking for to add a section for predefined relationships in our ontologies.
Designing for flexibility
While predefined relationships worked well for the coreference use cases, it wasn't what the dependency parsing users expected. In fact, they did not want to be forced to define their relationships at all because, in most cases, they were using the same names for NER annotations as they were for relationships (ie. "direct object" could be both an NER annotation and a relationship between two NER annotations). They didn't want to have to set up two separate tools in the ontology that had the same definition. I needed to figure out how to give users a way to create relationships on the fly inside the editor.
After doing some competitive analysis on similar text editors in the space and conducting more user tests on different interactions, I landed on one solution that would work well for both long and short text samples: Users could also create relationships by alt + clicking an annotation and then clicking the destination annotation.
Finishing touches
Once the overall UX and interactions were figured out, it was important to think through any edge cases and solve for those. Mainly - especially for coreference use cases where the text documents are long - how do we show relationships that extend outside of the page in view?After experimenting with a couple ideas and looking to other products for inspiration, I finally landed on this hover interaction for relationships that run outside of a given viewport.
Evolving to support conversational text
Around the Spring of 2022 (about one year prior to the LLM boom), we started getting more requests to support conversational text labeling. We learned that customers wanted to upload their previously exported chats with their internal chatbots so that they could label and further train them.
Because these files were JSON files instead of traditional .txt files, this would require a new project type - but technically they would support all of the same rules defined by our original text editor: NER annotations, relationships, and classifications.
There would however need to be a few performance adjustments based on some unique aspects of both conversational files and what we knew our customers were trying to do with them:
Adjustments
Most of these improvements were pretty straightforward. The one that I "fell" into (after first seeing the monster that the original design manifested on a large monitor), but was perhaps the most important, was the decision to set a max width of 640 px for the text and place it within a container centered in the middle of the editor canvas.
Message based classifications and LLM finetuning
As LLMs started surging, we learned that people want to export their conversations with these models and label them at a message level for finetuning purposes (similar to ChatGPT message level downvote with explanation).
Configuration
The easier part of the process was making a small change to our UX for creating global (asset level) classifications to confine them to message only. A small control within the UI for classification configuration made sense.
Message based classifications in editor
Here’s the first iteration of the designs for message based classifications within the editor. I was originally optimizing for horizontal real estate for the classifications.
Issues with relationships
Everything looked great...until I started layering in relationships. They overlapped with the message based classification containers and would surely cause labeling problems.
Consolidating space / final solution
After much thought and other explorations, I finally found that having the message text exist on the left with the message classifications on the right solved most overlap issues with relationships.I had some concerns with classification titles and options being long, but was okay with the compromise of them wrapping and maybe being taller vertically than they would have been in the first iteration.
.png)
