learnkdb

System Features

File Parsing

Examples demonstrating how to ingest and parse file data

Overview

Much of the data loaded into kdb+ arrives as text files — CSV extracts, vendor reference files, logs. q has a small set of tools for reading them and turning them into typed lists and tables:

  • read0 reads a text file as a list of strings, one per line.
  • vs ("vector from scalar") splits a string on a delimiter; sv joins strings back together.
  • Tok ($ with an uppercase type character, e.g. "J"$) parses strings into typed values.
  • 0: with a list of type characters parses a delimited file straight into a table.

Key Things to Know

Reading lines and splitting them:

/ Write a small file to work with: 0: with a list of strings writes one line per string
`:pairs.txt 0: ("Binance_BTC-USDT";"Bybit_BTC-USDC");

/ read0 returns one string per line
read0 `:pairs.txt
"Binance_BTC-USDT"
"Bybit_BTC-USDC"
/ vs splits a string on a delimiter
"_" vs "Binance_BTC-USDT"
"Binance"
"BTC-USDT"

Parsing text into typed values with Tok — the uppercase type character parses rather than casts:

"J"$("10";"25";"7")
"F"$"3.25"
10 25 7
3.25

Loading a CSV file with a header row into a table: the left argument gives one type character per column, and enlist "," says the file is comma-delimited with column names in the first row.

`:trades.csv 0: ("sym,price,size";"AAPL,150.25,100";"MSFT,310.5,200")
("SFJ"; enlist ",") 0: `:trades.csv
sym  price  size
----------------
AAPL 150.25 100
MSFT 310.5  200

Practice

The questions in this topic apply these tools to small parsing problems: decoding a compact text format into numbers, turning delimited reference data into a table and querying it, and inspecting the tables in a session.

Sections