Perspective <> TDEngine Integration

Real-Time Data Visualization with TDengine and Perspective

Introduction

This guide demonstrates how to integrate TDengine, a high-performance time-series database, with Perspective, a powerful real-time data visualization library. By combining these technologies, you can create a scalable platform that streams and visualizes fast-moving data in a browser-based interactive dashboard.

Key Steps in This Guide:

  1. Install TDengine Client: Set up the TDengine Python client.
  2. Run TDengine in Docker: Deploy a TDengine Docker container.
  3. Create a Virtual Environment: Configure a Python virtual environment with necessary dependencies.
  4. Generate Data: Implement a script to simulate real-time data ingestion into TDengine.
  5. Run the Perspective Server: Stream data from TDengine to a Perspective viewer via a WebSocket.
  6. Visualize Data: Embed and configure a Perspective viewer in an HTML page.

Overview

TDengine: A High-Performance Time-Series Database

TDengine is an open-source, scalable time-series database designed for handling large volumes of time-series data. It excels in real-time analytics use cases such as IoT, finance, and telecommunications.

Key Features:

  • High Throughput: Supports millions of inserts per second.
  • Compression: Reduces storage requirements by up to 90%.
  • SQL Compatibility: Uses a SQL-like query language.
  • Scalability: Distributed architecture for handling large datasets.

Perspective: A Powerful Real-Time Visualization Library

Perspective is an interactive analytics and data visualization component built for high-performance, real-time rendering of large datasets.

Key Features:

  • WebAssembly & Apache Arrow: Ensures fast processing and memory efficiency.
  • Custom HTML Element (<perspective-viewer>): Embeds dynamic tables and charts in web applications.
  • Real-Time Updates: Automatically refreshes as new data arrives.
  • Multi-Language Support: Integrates with Python, Node.js, and Rust.

Together, TDengine and Perspective enable real-time analytics for applications in monitoring, financial markets, and IoT.

Architecture

The architecture includes:

  • TDengine Database: Stores and manages time-series data.
  • Python Producer Script (producer.py): Simulates real-time data ingestion.
  • Perspective Server (perspective_server.py): Streams data from TDengine to Perspective.
  • Perspective Viewer (prsp-viewer.html): Displays the data interactively in a browser.

Getting Started

Please start by cloning our Perspective Examples Github repo:

git clone https://github.com/ProspectiveCo/perspective-examples.git

# navigate to the tdengine dir
cd perspective-examples/examples/tdengine

1. Install TDengine Client

Run the install.sh script to install the TDengine Python client libraries:

./install.sh

Verify the installation:

ls -l tdengine-client/driver/
echo $LD_LIBRARY_PATH

Ensure LD_LIBRARY_PATH includes the client library path.

2. Start a TDengine Docker Container

Deploy TDengine in Docker using the docker.sh script:

./docker.sh

To populate the container with benchmark data:

./docker.sh --benchmark

3. Create a Virtual Environment

Activate and install dependencies:

source venv/bin/activate
pip install -r requirements.txt

4. Generate Data in TDengine

Run the producer.py script to simulate real-time stock trade data:

python producer.py

Key Features:

  • Generates stock trade data with timestamps, tickers, and prices.
  • Inserts data into TDengine every 250ms.

Example table schema:

create_table = """
CREATE TABLE IF NOT EXISTS stock_values (
    timestamp TIMESTAMP,
    ticker NCHAR(10),
    client NCHAR(10),
    open FLOAT,
    high FLOAT,
    low FLOAT,
    close FLOAT,
    volume INT UNSIGNED,
    date TIMESTAMP
)
"""

5. Run the Perspective Server

Start the Perspective server to stream data from TDengine:

python perspective_server.py

Key Components:

  • Reads Data: Queries TDengine for the latest records.
  • Manages Perspective Table: Stores and updates real-time data.
  • Streams Data via WebSocket: Publishes updates to connected clients.

Example code snippet:

def updater():
    data = read_tdengine()
    table.update(data)

tornado.ioloop.PeriodicCallback(updater, 250).start()

6. Open the Perspective Viewer

Launch prsp-viewer.html in a browser to visualize data in real-time:

open prsp-viewer.html

Viewer Configuration:

  • Uses <perspective-viewer> component.
  • Connects to WebSocket at ws://localhost:8080/websocket.

Example HTML snippet:

<perspective-viewer id="prsp-viewer" theme="Pro Dark"></perspective-viewer>

Code Explained

Perspective Server (perspective_server.py)

  • Creates Perspective Server
perspective_server = perspective.Server()
  • Defines Table Schema
schema = {
    "timestamp": "datetime",
    "ticker": "string",
    "client": "string",
    "open": "float",
    "high": "float",
    "low": "float",
    "close": "float",
    "volume": "float",
    "date": "date",
}
  • Streams Data to Clients
app = tornado.web.Application([
    (r"/websocket", perspective.handlers.tornado.PerspectiveTornadoHandler, {"perspective_server": perspective_server}),
])
app.listen(8080)
tornado.ioloop.IOLoop.current().start()

Commercial Version: Prospective.co

Prospective offers a commercial version of Perspective with additional features:

  • Built-in TDengine Integration
  • Advanced Dashboards
  • Enterprise Security & Scalability

For a free trial, email hello@prospective.co with the subject "TDengine Trial".

Conclusion

By following this guide, you've learned how to:

  1. Install and configure TDengine.
  2. Set up a Docker container for TDengine.
  3. Generate real-time data with Python.
  4. Stream data into Perspective via WebSockets.
  5. Visualize and interact with data using <perspective-viewer>.

This integration enables real-time analytics for industries like finance, IoT, and monitoring.

For further details, refer to:

Happy coding!