Scout Data – PJ Paydays

Historical End-of-Day Price Data

a look at online data vendors – spoiler alert: we will use Tiingo
explore the Tiingo API
examine API responses and status codes
define common data types for our application
build a client to query the Tiingo API

Vendors

Historical Price Data Vendors

Google this: “end of day stock data”

eoddata.com

lots of markets, varying prices from free to $450 annual
confusing for entry level programming
designed to integrate with commercial platforms
rather expensive for our purposes

eodhistoricaldata.com

free            :: 20 api calls /day
$20-$80 / month :: 100,000 api calls /day

api.tiingo.com

free        :: 500 calls/hr,  20K /day,   5 GB /month
$10 / month :: 20K calls/hr, 150K /day, 100 GB /month
$50 / month :: 40K calls/hr, 400K /day, 400 GB /month

We will be working with Tiingo
– the free account provides generous calls for our development
– $10 / month handles daily updates when fully implemented
– the API is simple and well documented

Tiingo API

The Tiingo API

Preliminary:
Tiingo requires users to register an account
– register for free account @ api.tiingo.com
– provide Username, E-mail, Password
– receive authentication key (API token)

The key looks something like:
“b2f9c56d8554dea1e733c1df0af9f4e0519617e3”
(don’t try to use this one – it is bogus, for example only)

In this discussion, we will refer to this key as MyToken

After reviewing the documentation, we see we can define some variables
to help with our documentation and discussion

Name	Description	Value
Tiingo	the common URL of the API calls	https://api.tiingo.com/tiingo/daily
ticker	a stock symbol	AAPL, GM, SP, etc.
startDate	first data date	yyyy-mm-dd
endDate	last data date	yyyy-mm-dd (optional)
format	returned data format	‘JSON’ (default) \| ‘csv’
resampleFreq	data frequency	‘daily’ (default) \| ‘weekly’ \| ‘monthly’
token	access key	MyToken

So, what data is available?

# Meta Data
https://api.tiingo.com/tiingo/daily/ticker?token=MyToken

Tiingo/ticker?token=MyToken

# Latest Price
https://api.tiingo.com/tiingo/daily/ticker/prices?token=MyToken

Tiingo/ticker/prices?token=MyToken

# Historical Prices
https://api.tiingo.com/tiingo/daily/ticker/prices
  ?startDate=2021-07-01&endDate=2021-07-02&token=MyToken    

Tiingo/ticker/prices
  ?startDate=startDate&endDate=endDate&token=MyToken

Examples (using ticker ‘GM’ – General Motors Company)

# Meta Data

(JSON)
Tiingo/GM?token=MyToken

{"endDate": "2021-07-06", "exchangeCode": "NYSE", 
 "description": "General Motors is ...", 
 "ticker": "GM", "name": "General Motors Company", 
 "startDate": "2010-11-18"}

(CSV)
Tiingo/GM?format=csv&token=MyToken

ticker,name,exchangeCode,description,startDate,endDate
GM,General Motors Company,NYSE,"General ...",2010-11-18,2021-07-08

Note: description is not interesting (blah, blah, blah)
we want: 
ticker       - vendor's internal name name for the ticker
startDate    - first available date from vendor
endDate      - last available date from vendor
name         - descriptive name, not just the ticker symbol
exchangeCode - maybe. Useful if you are filtering by exchange

# Latest Price

(JSON)
Tiingo/GM/prices?token=MyToken

[{"adjClose":56.06,"adjHigh":56.585,"adjLow":54.86,"adjOpen":55.3,
  "adjVolume":14087371,"close":56.06,
  "date":"2021-07-08T00:00:00+00:00","divCash":0.0,"high":56.585,
  "low":54.86,"open":55.3,"splitFactor":1.0,"volume":14087371}
]

(CSV)
Tiingo/GM/prices?format=csv&token=MyToken

,adjClose,adjHigh,adjLow,adjOpen,adjVolume,close,date,divCash,high,low,open,splitFactor,volume
0,56.06,56.585,54.86,55.3,14087371,56.06,2021-07-08T00:00:00+00:00,0.0,56.585,54.86,55.3,1.0,14087371

# Historical Prices

# general form
Tiingo/ticker/prices
  ?startDate=startDate
  &endDate=endDate            (optional = latest available)
  &format=format              (JSON (default) | csv)
  &resampleFreq=resampleFreq  (daily (default) | weekly | monthly)
  &token=MyToken

# daily price data – July 1 and 2, 2021

(JSON)
Tiingo/GM/prices
  ?startDate=2021-07-01&endDate=2021-07-02
  &token=MyToken

[{"date":"2021-07-01T00:00:00.000Z",
  "close":59.11,"high":59.73,"low":58.775,"open":59.44,"volume":11752117,
  "adjClose":59.11,"adjHigh":59.73,"adjLow":58.775,"adjOpen":59.44,"adjVolume":11752117,
  "divCash":0.0,"splitFactor":1.0},
 {"date":"2021-07-02T00:00:00.000Z",
  "close":58.96,"high":59.32,"low":58.52,"open":59.11,"volume":9699563,
  "adjClose":58.96,"adjHigh":59.32,"adjLow":58.52,"adjOpen":59.11,"adjVolume":9699563,
  "divCash":0.0,"splitFactor":1.0}
]

(CSV)
Tiingo/GM/prices
  ?startDate=2021-07-01&endDate=2021-07-02
  &format=csv
  &token=MyToken

date,close,high,low,open,volume,adjClose,adjHigh,adjLow,adjOpen,adjVolume,divCash,splitFactor
2021-07-01,59.11,59.73,58.775,59.44,11752117,59.11,59.73,58.775,59.44,11752117,0.0,1.0
2021-07-02,58.96,59.32,58.52,59.11,9699563,58.96,59.32,58.52,59.11,9699563,0.0,1.0

API Responses

Tiingo API query response and status codes

Meta Data Requests

  Tiingo/ticker?token=MyToken   { &format=csv }

  Success( ticker 'GM' )
  ====================================================
  JSON
    statusCode = '200', statusText = 'OK'
    response   = {"endDate": "2021-07-16", "
                  exchangeCode": "NYSE",
                  "description": "General Motors ...",
                  "ticker": "GM",
                  "name": "General Motors Company",
                  "startDate": "2010-11-18"}
  CSV
    statusCode = '200', statusText = 'OK'
    response
    :::::<begin>:::::
    ticker,name,exchangeCode,description,startDate,endDate
    GM,General Motors Company,NYSE,"General Motors ...",2010-11-18,2021-07-16
        <blank line>
    :::::<end>:::::
  =========================================================

  Failure( bad ticker 'GMxxxxx' )
  =========================================================
  JSON
    statusCode = '404', statusText = 'Not Found'
    response   = {"detail":"Not found."}

  CSV
    statusCode = '404', statusText = 'Not Found'
    response
    :::::<begin>:::::
    None
    :::::<end>:::::
  =========================================================

Latest Price

  Tiingo/ticker/prices?token=MyToken   { &format=csv }

  Success( ticker 'GM' )
  =========================================================
  JSON
    statusCode = '200', statusText = 'OK'
    response   = [{"adjClose":55.46,"adjHigh":57.48,
                   "adjLow":55.37,"adjOpen":57.48,
                   "adjVolume":13611414,"close":55.46,
                   "date":"2021-07-16T00:00:00+00:00",
                   "divCash":0.0,"high":57.48,"low":55.37,
                   "open":57.48,"splitFactor":1.0,
                   "volume":13611414}]

  CSV
    statusCode = '200', statusText = 'OK'
    response
    :::::<begin>:::::
    ,adjClose,adjHigh,adjLow,adjOpen,adjVolume,close,date,divCash,high,low,open,splitFactor,volume
    0,55.46,57.48,55.37,57.48,13611414,55.46,2021-07-16T00:00:00+00:00,0.0,57.48,55.37,57.48,1.0,13611414
        <blank line>
    ::::::::::
  =========================================================

  Failure( bad ticker 'GMxxxxx' )
  =========================================================
  JSON
    statusCode = '404', statusText = 'Not Found'
    response   = {"detail":"Error: Ticker 'GMXXXXX' not found"}

  CSV
    statusCode = '200', statusText = 'OK'
    response
    :::::<begin>:::::
    Error: Ticker 'GMXXXXX' not found
    :::::<end>:::::
  =========================================================

Historical Prices

  Tiingo/ticker/prices
    ?startDate=startDate
    &endDate=endDate   *optional
    &token=MyToken     ( &format=csv )

  Success( ticker 'GM' )
  =========================================================
  JSON
    statusCode = '200', statusText = 'OK'
    response   = [{"date":"2021-07-01T00:00:00.000Z",
                  "close":59.11,"high":59.73,"low":58.775,
                  "open":59.44,"volume":11752117,
                  "adjClose":59.11,"adjHigh":59.73,
                  "adjLow":58.775,"adjOpen":59.44,
                  "adjVolume":11752117,"divCash":0.0,
                  "splitFactor":1.0},
                  {"date":"2021-07-02T00:00:00.000Z",
                  "close":58.96,"high":59.32,"low":58.52,
                  "open":59.11,"volume":9699563,
                  "adjClose":58.96,"adjHigh":59.32,
                  "adjLow":58.52,"adjOpen":59.11,
                  "adjVolume":9699563,"divCash":0.0,
                  "splitFactor":1.0}]

  CSV
    statusCode = '200', statusText = 'OK'
    response
    :::::<begin>:::::
    date,close,high,low,open,volume,adjClose,adjHigh,adjLow,adjOpen,adjVolume,divCash,splitFactor
    2021-07-01,59.11,59.73,58.775,59.44,11752117,59.11,59.73,58.775,59.44,11752117,0.0,1.0
    2021-07-02,58.96,59.32,58.52,59.11,9699563,58.96,59.32,58.52,59.11,9699563,0.0,1.0
        <blank line>
    :::::<end>:::::
  =========================================================

  Failure( bad ticker 'GMxxxxx' )
  =========================================================
  JSON
    statusCode = '404', statusText = 'Not Found'
    response   = {"detail":"Error: Ticker 'GMXXXXX' not found"}

  CSV
    statusCode = '200', statusText = 'OK'
    response
    :::::<begin>:::::
    Error: Ticker 'GMXXXXX' not found
    :::::<end>:::::
  =========================================================

  Failure( bad start date '2021-13-01' )
  =========================================================
  JSON
    statusCode = '400', statusText = 'Bad Request'
    response   = ["Error: Start date format was not correct. Must be in YYYY-MM-DD format."]

  CSV
    statusCode = '200', statusText = 'OK'
    response
    :::::<begin>:::::
    Error: Start date format was not correct. Must be in YYYY-MM-DD format.
    :::::<end>:::::
  =========================================================

  Failure( bad end date '2021-07-0x' )
  =========================================================
  JSON
    statusCode = '400', statusText = 'Bad Request'
    response   = ["Error: End date format was not correct. Must be in YYYY-MM-DD format."]

  CSV
    statusCode = '200', statusText = 'OK'
    response
    :::::<begin>:::::
    Error: End date format was not correct. Must be in YYYY-MM-DD format.
    :::::<end>:::::
  =========================================================

  Failure( start date is after end date )
  =========================================================
  JSON
    statusCode = '200', statusText = 'OK'
    response   = []

  CSV
    statusCode = '200', statusText = 'OK'
    response
    :::::<begin>:::::
    date,close,high,low,open,volume,adjClose,adjHigh,adjLow,adjOpen,adjVolume,divCash,splitFactor
        <blank line>
    :::::<end>:::::
  =========================================================

Data Types

MetaData, Daily Prices, and Intraday Prices

Tiingo Client

A client interface to the Tiingo API

Binary vs SQL

Binary vs SQL data storage

Binary Store

Binary Store

SQL Store

SQL Store