OpenDataDSL
Data Loaders
Load any file or API into the platform, without writing a script
Build a loader for a broker file, an exchange workbook, a JSON API or a table on a web page step by step on screen, map its columns to your fields, and run it on a schedule.
The Data Loading Challenge
The Challenge
A lot of the data a trading business relies on does not come from a market data vendor. It arrives as:
- A broker's end of day settlements as a CSV file
- An exchange's settlement workbook
- Readings from a wind farm's JSON API
- Prices published in a table on a web page
Getting each one into the platform used to mean writing a script by hand: reading the file, working out the date formats and decimal commas, deciding which rows to leave out, and making sure a rerun does not load everything twice.
Every new source waits for someone who can write that script.
Our Solution
Data Loaders lets you build that loader step by step, on screen. Point it at the source and the platform reads a sample exactly as a run will, shows you every column, and lets you map each column to the fields of an event type with a line from one to the other.
Choose where the events go and how often to load them. Deploying creates a workflow of standard actions and a process that runs it, with a report of every run.
Fusion AI can propose the event type and the whole mapping from the sample, asking you about anything only you can decide.
Data Loaders: From a File or API to Events, With No Code
Key Benefits
Bring every file and API your business relies on into the platform, built and run by the people who know the data.
Any File or API
Six kinds of source.
CSV and text files, JSON APIs, XML documents, Excel workbooks and tables on web pages, read from a URL, or your own ODSL for anything else. Pick a table on a web page by clicking it.
See the Data First
A real sample, examined for you.
The platform reads the first rows exactly as a run will, and shows every column with what it holds: numbers, dates, text, how full it is, and examples of its values.
Map Columns Visually
Drag a column onto a field.
Columns on the left, the event's fields on the right, joined by a line. Set the data type, the date format, a decimal comma, a scale, empty values and a value map for each field.
Help From Fusion AI
A proposal from your sample.
Fusion AI reads the sample and proposes the event type, the mapping and the event id, and asks with buttons when a choice is yours. Describe a change to a field's text in your own words and it writes it.
Safe to Run Again
No duplicates.
Each event's id is built from its time and the columns that make a row unique, so loading a file again replaces its events instead of adding copies.
Scheduled and Reported
Every run recorded.
Run on weekdays, every day, every hour or a custom schedule, with a holiday calendar. Each run's report shows the rows read, the events saved and the rows left out, and a failed run raises an alert.
How Data Loaders Work
Build It, Step by Step
Extract: choose the kind of source and its URL, which can contain the date being loaded, variables and secrets such as an API key.
Examine: the rows of the sample and what each column holds.
Event type: choose one, define a new one from the columns, or use none.
Map: join each column to a field, and preview the events the sample makes.
Destination: the master data record and event list the events go to, the event id, and the dataset id for monitoring.
Deploy and Run It
When every check passes:
Deploy: saves a workflow of the standard extract, transform, load and report actions, and a process that runs it.
Schedule: weekdays at 18:30 UK time, every hour, or any cron schedule, in the time zone you choose.
Run now: load today, or any earlier date, and see the run report straight away.
Curves and series: make event timeseries, forward curves and option chains from the events, once, on the same record.
Loading a table from a web page: click the table, and the platform reads it the same way every run.
Real-World Applications
Data that energy and commodity teams load with Data Loaders.
Broker End of Day Settlements
Two brokers send a semicolon separated file of TTF, THE, NBP and PEG settlements every evening. A loader saves them as events every weekday at 18:30, with the decimal commas and contract names handled.
Exchange Settlement Workbooks
Read the power settlements from a sheet of an Excel workbook, leave out the total rows, and build a forward curve for each day from the events.
Weather at Every Wind Farm
Call a weather API once for each wind farm's latitude and longitude, with the API key kept in the secret store, and load the readings every hour.
Prices From a Web Page
Pick the table of metal prices on a public web page, turn the date and AM or PM marker into the event time and name, and load it every morning.
Integration with Platform Capabilities
Data Loaders bring in the data the rest of the platform uses
From Hand-Written Scripts to Loaders Anyone Can Build
Data Loaders turns loading a new file or API from a development task into a few minutes on screen, with the sample in front of you at every step and Fusion AI to help with the mapping.
The people who know the data build the loaders, every run is scheduled, reported and safe to repeat, and a new source no longer waits for someone who can write the script.
See Your Files Loaded in Minutes
Book a demo and we will build a loader for one of your own files or APIs while you watch.
Get Started TodayValidation
How about a demo?
Our team is here to find the right solution for you, contact us to see this in action.
Fill out your details below and somebody will be in contact with you very shortly.