Innovators Featured in Article on Bridging Power Company Data and AI
VIA is proud to be featured in an article by our partner and Europe’s largest energy utility, Enel. The article, which is published as a success story on Enel’s website, examines data’s critical role in powering the energy transition and the challenges many power companies face in tapping into this precious resource. Highlighting the partnership between VIA and Enel X, Enel’s demand response business line, Enel explores the innovations, including our TAC™ platform and SWEET™ technology, that are helping it bridge data with AI to overcome these barriers. With SWEET, Enel X is able to automate data cleaning and wrangle thousands of data files from Enel X customers in just minutes — which otherwise may have taken hours for a data scientist to manipulate manually.
We look forward to continued collaboration with Enel X as we pave the way to a clean energy future, together. As Kate Ravanis, COO of VIA, concludes in the piece: “Anyone can get lucky and make something new once. For us, we strive to find a faster, smarter, cleaner way of doing everything, all the time.”
To read the full article by Enel X, visit their website here.
https://www.solvewithvia.com/wp-content/uploads/2021/11/EnelandVIA.png6271201viahttps://www.solvewithvia.com/wp-content/uploads/2018/02/VIA-Logo-header-300x101.pngvia2021-11-09 08:30:102022-11-30 05:24:21Europe’s Largest Utility Spotlights Partnership with VIA to Lead Energy Transition
For the third installment of our blog series, Let’s TAC™ About It, we’d like to introduce you to SWEET, a built-in function on VIA’s TAC™ platform. So, are you ready to TAC™ about it?
SWEET stands for Smart Wrangling Engine for Extraction and Transformation. Check out the video (or the transcription that follows) to learn how we use AI to wrangle data!
Below is a transcription of the “Introduction to SWEET: AI for Data Wrangling” video.
Analysts need to transfer data from a spreadsheet to a database, often known as data wrangling. The analyst usually identifies a rule or multiple rules (such as: column x is data, row 5 is a header and can be discarded, and so on). The analyst then writes code to execute that rule. This works well when rules are easily identifiable. In most cases, however, this is incredibly time consuming.
Data scientists spend more time wrangling and cleaning data than on analysis and AI. The problem with that is analytics insights are of the highest value, but get the least amount of resources. The big leap in AI is being able to process information without humans writing all the rules.
As an example, computer vision is used to identify a dog. Think of all the varieties of dogs and all the possible variations in context that those dogs could be in. There isn’t an army of people large enough to write rules to identify a random dog in a random photo.
And yet, AI can.
VIA’s approach to data wrangling is to use some of the exact same AI algorithms used in image recognition. This works across a much wider variety of contexts and spreadsheet or file formats. Let’s take a look at how SWEET works.
Here’s a spreadsheet. What SWEET is going to do is use a number of different machine learning algorithms to automate the process of getting the information into a database format.
The first model uses machine vision to map out the spreadsheet. Purple represents blank space, green is headings, yellow is actual data. Once that model has run, there’s a second algorithm that takes a look at the content.
The second algorithm skips over the purple. It looks at the green (which is the headers) to know which column to write where in the database. Finally, it would take a look at the yellow area to write the data to the database.
A third machine learning algorithm determines which column is derived from other parts of the sheet. For example, a total column is just the sum of the other columns and may not be necessary to write to the database. The third model separates these derived columns from the raw data.
In this example, the ACCOUNT column turns out to be the total of the other columns added up. It could be difficult for a human to understand immediately, but one of our models does this instantly.
SWEET’s approach works irrespective of the format. The model doesn’t have to be re-trained when it comes across spreadsheets that are new or in different formats.
So, what’s new and different here? AI algorithms have been evolving quickly. Many of the models that we implemented didn’t exist just a few years ago.
The other insight is that we broke the “convert this spreadsheet into a db” problem into multiple steps and have a different AI algorithm for each step.
Combined, SWEET, a built-in function in VIA’s TAC™ ingestion engine, helps make processes that used to take analysts days to do manually and makes them instantaneous.
Update: July 31, 2020
The image below shows the steps SWEET takes to transform raw files into a standard format.
https://www.solvewithvia.com/wp-content/uploads/2020/07/Lets-TAC-About-It-3.jpg6221438viahttps://www.solvewithvia.com/wp-content/uploads/2018/02/VIA-Logo-header-300x101.pngvia2020-07-22 08:00:292022-11-30 05:24:50Introducing SWEET: AI for Data Wrangling
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.