Nexis Data Lab is an exciting first for LexisNexis. Historically a content provider, LexisNexis recognized an opportunity to create something a little different for students, professors and other academic researchers: A data research and analysis tool that enables them to curate their own datasets and dive into analytics faster – all on a single platform. No more offloading large quantities of raw data to your laptop and no more time-consuming data wrangling. Because with Data Lab, researchers can search and refine datasets in the Data Lab UI, applying a series of post- and pre-search filters, and then analyze and visualize potentially thousands of documents in minutes using Python or R within a secure Jupyter Notebook environment.
Developing Nexis Data Lab
Of course, creating this type of unique solution wasn’t without its share of technical logistics to overcome. However, as we recently found out from a developer who worked on Nexis Data Lab, the superhuman-like ability to solve for those logistics efficiently and creatively came from the people working on it.
“I’m fortunate that I have a very talented development team that I work with,” said consulting software engineer Kevin Haverlock. We sat down with Kevin to discuss his role in developing Nexis Data Lab the challenges he and his team had to overcome along the way, and what he enjoyed most about the experience. He was quick to heap praise on his teammates when it came to discussing how they tackled technical hurdles and developed creative solves for technical dilemmas. In fact, he said it was the people he enjoyed most about working on Data Lab.
When pressed on what else he enjoyed, he answered, “the technology.” He continued, “In your software career, it’s not very often that you get the opportunity to do greenfield development, where you’re actually building something that nobody has seen before. So, in the sense of Nexis Data Lab, that’s really what that is. It’s really a new class of software for LexisNexis in that it allows customers to do their own development and find their own insights.”
Kevin went on to explain that an opportunity to work on greenfield development (i.e., creating a digital product or solution entirely from scratch) is enticing to all developers out there. He also reflected on the fact that while greenfield development is exciting and somewhat scary at the same time, the “what-if” factor of it all was multiplied exponentially by the reality that this kind of product was entirely new to LexisNexis.
“If you look at LexisNexis, we’re a big provider of data,” said Kevin. “We provide legal content, news content, company information – we’re a content provider. Two ways customers get access to that content is through APIs, so they can call our APIs and get our data back. The other way is through the applications, where we pull the data ourselves or provide a UI that will display insights into the data.
Data Lab takes a different approach, where we actually go and build a dataset for you based on your search. So we’ll package it up as a dataset. There’s also an analysis environment. So, you as a researcher or a student, you can write your own code to get your own insights into the data. It’s very much an open-ended tool that puts the user in control of how they want to explore this dataset that they’ve created. That’s the exciting thing about Nexis Data Lab.”
“Otherwise, you as a researcher, when you use one of our APIs, you download the content you want to analyze, you build that dataset locally on your laptop, and then you’re finally able to start doing your analytics. Nexis Data Lab makes the work of finding and analyzing data so much more efficient. It streamlines dataset building, and then lets you seamlessly jump into analyzing your dataset using Jupyter Notebook. And the user never has to leave the Data Lab environment.”
How Nexis Data Lab got its start
Given that this kind of solution was (at the time) an out-of-the-box idea for LexisNexis, there had to be an interesting origin story behind it. When asked, Kevin didn’t shy away from sharing how the idea for Nexis Data Lab presented itself.
“Nexis Data Lab has some basis around Nexis Uni®,” said Kevin. “If you look at our UI, it looks very similar. Nexis Uni allows students to do research on news and Nexis Data Lab still supports that. We have taken it one step further by allowing a researcher to use a Jupyter Notebook to write their own algorithms across the news content. As an example, we have seen students use it to study the use of polarizing words in controversial news coverage, word usage over time, or even sentiment analysis of a subject. We have been excited to see what researchers will come up with”
Some favorite features of Nexis Data Lab
Next, we asked him about the specific features of Data Lab that Kevin likes the most. “The beauty of it is that you log in and do your search and get back a whole set of search results,” Kevin responded. “For example, if you’re searching for COVID and NC, you’ll get a whole ton of news content that include mentions of COVID and NC. From there, you can customize your dataset by applying filters. You press ‘Create Workspace’ button, and then you open a Jupyter Notebook environment where you can analyze the data yourself in our environment.”
The team effort behind Nexis Data Lab
Hearing Kevin talk during our sit-down, it was easy to imagine the sheer amount of collaboration across teams to figure out the UI and backend functionality. When we asked him about it, he was quick to respond that this kind of development can’t happen in a vacuum if you want a quality product.
“Oh, there were a lot of whiteboarding sessions, and we worked very closely with the product team. There was an agreed-upon vision of the final product, and then multiple teams collaborated on how to build and put together the technology pieces necessary to make it happen.”
And aside from the product team, Kevin and his fellow developers worked with cloud engineers to poke holes, find answers and iterate until Nexis Data Lab started to take shape.
“There are all sorts of little problems that you have to overcome along the way,” says Kevin. “I’m going to make all these API calls to go and get the data, then I’m going to copy the data into a data store. How do I do that cheaply? What is the right technology choice so that we do this in a way that’s economical (e.g., making the most efficient use of cloud resources). Things like that steered some of our technology decisions.”
Kevin also pointed out the importance of working with the LexisNexis security team. “A product that allows customers to execute code has a lot of security questions. So, we worked closely with the security team to go and vet what we were doing along the way.”
Nexis Data Lab: A worthwhile endeavor
For Kevin and his team, they always saw the potential of Nexis Data Lab and the benefits it could provide to academic researchers. But everything was little more than theory until they had the chance to get Nexis Data Lab up on its feet.
“When you’re doing software development, you get these little pieces of success along the way,” said Kevin. “’Oh, hey the UI came up. Oh, hey I was able to populate the database,’ and that’s a great feeling. But it really isn’t until that moment when you see that thread where everything works together end to end. There is a huge sensation of relief because you realize how well this is going to work. It was really when we were able to do a search and see a workspace populate and launch a Jupyter Notebook for the first time that we knew we had something.”
And Nexis Data Lab has been off to the races ever since that moment. As Kevin would go on to explain, the team eventually moved Data Lab into discovery with customers. They showed it to graduate and undergraduate students to let them experiment with it and give them their feedback, all of which helped Nexis Data Lab become the product that it is today.
You can learn more about Nexis Data Lab here.