Transforming Streaming Data into OHLC Format with R and Lightstreamer

Introduction

In this article, we will explore how to transform streaming data from a Lightstreamer client in R into an xts object containing Open, High, Low, and Close (OHLC) values. We will go through the process step by step, explaining each part of the code and highlighting key concepts.

Background

Lightstreamer is a real-time communication platform that enables bidirectional communication between clients and servers over the web. In this context, we are using Lightstreamer as a client to receive streaming data from a server. The data stream contains tick data in a proprietary format, which we will need to transform into OHLC values.

Reproducing the Streaming Data

To start, we can reproduce the Lightstreamer tick data visible in the R console by creating a loop that sends the proprietary data format to the file ‘/lightstreamer_v3/a10.txt’.

counter <- 1 # Start position of counter 
sink (file = '/lightstreamer_v3/a10.txt', append = TRUE, split = TRUE) # store data to file. Adjust to your path.

repeat {
    counter <- counter + 1  # Handle internal loop
    cat ("<<  Preamble: preparing push", "\n") # Printouts to R console.
    cat ("<<  Preamble: preparing push","\n")
    cat ("<<  Preamble: preparing push","\n")  
    cat ("<<  Preamble: preparing push","\n")
    cat ("<<","\n")
    cat ("<<","\n")
    cat ("<<","\n")
    cat ("<<",""\n")
    cat ("<<",""\n")
    cat ("<<",""\n")
    cat ("<<",""\n")
    cat ("<<",""\n")
    cat ("<<",""\n")
    cat ("<<",""\n")
    cat ("<<", "U,3,1,20:00:33|3.04|0.0|2.41|3.67|3.03|3.04|#|#|$","\n") # tick-data
    cat ("<<","\n")
    cat ("<<",""\n")
    cat ("<<",""\n")
    cat (" 7", "\n")
    cat (" PROBE", "\n")
    if (counter >= 10000) { # Counter break.
        break 
     }
}

sink ()

Understanding the Data Format

The data format is a proprietary structure that contains multiple fields:

U: The type of message (in this case, a tick).
3: The sequence number of the message.
1: An indicator that the message is complete.
20:00:33|3.04|0.0|2.41|3.67|3.03|3.04|#|#|$: A timestamp and a series of values representing:
- The current price (3.04).
- The change in price (0.0).
- The minimum bid (2.41).
- The maximum bid (3.67).
- The last traded ask (3.03).
- The last traded bid (3.04).
- The open value (not specified in the given format, but assumed to be 3.04 based on the timestamp).

Transforming Data into OHLC Format

To transform this data into an xts object containing Open, High, Low, and Close values, we can use a combination of regular expressions and the xts package.

Step 1: Extract Timestamp and Values

First, we need to extract the timestamp and values from each message. We can do this by using regular expressions.

# Load necessary libraries
library(xts)

# Define regular expression pattern
pattern <- "(20:[0-9]{2}:[0-9]{2}:[0-9]{2}|[0-9]{1,3}.[0-9]{2,3})\\|[0-9]{1,4}:([0-9]{1,4}):([0-9]{1,6})"

# Function to extract timestamp and values
extract_data <- function(data) {
    # Use regular expression pattern to match data format
    matches <- strsplit(data, pattern)[[1]]
    
    # Extract timestamp (first 8 characters)
    timestamp <- substr(matches[1], start = 5, stop = 13)
    
    # Extract values (next 7 characters)
    values <- matches[2]
    
    return(c(timestamp = timestamp, values = strsplit(values, "|")[[1]]))
}

Step 2: Parse Values and Create OHLC Object

Next, we need to parse the values into Open, High, Low, and Close (OHLC) objects.

# Function to parse values
parse_values <- function(values) {
    # Assume Open = Last Traded Ask for simplicity
    open_value <- values[3]
    
    # Use last traded ask as close value
    close_value <- values[4]
    
    # Set high and low based on price and change
    if (values[2] > 0) {  # Increase
        high_value <- max(open_value, values[5])
        low_value <- min(open_value, values[6])
    } else {  # Decrease
        high_value <- max(values[3], values[7])
        low_value <- min(values[2], values[8])
    }
    
    return(list(
            open = as.numeric(open_value),
            high = as.numeric(high_value),
            low = as.numeric(low_value),
            close = as.numeric(close_value)
        ))
}

Step 3: Create xts Object

Finally, we can create an xts object from the extracted data and OHLC values.

# Function to create OHLC xts object
create_ohlc_xts <- function(data, parsed_values) {
    # Create data frame with timestamp and OHLC values
    df <- data.frame(
        Time = data$timestamp,
        Open = parsed_values[1]$open,
        High = parsed_values[1]$high,
        Low = parsed_values[1]$low,
        Close = parsed_values[1]$close
    )
    
    # Convert data frame to xts object
    ohlc_xts <- xts(df, order.by = "Time")
    
    return(ohlc_xts)
}

Putting It All Together

Now that we have broken down the steps to transform streaming data into OHLC format with R and Lightstreamer, let’s put it all together in a single function.

# Combine functions into a single function
transform_data <- function() {
    # Read data from file
    data <- read.csv('/lightstreamer_v3/a10.txt', header = FALSE)
    
    # Parse values into OHLC object
    parsed_values <- sapply(strsplit(data, "|"), function(x) parse_values(x))
    
    # Create xts object from parsed values
    ohlc_xts <- create_ohlc_xts(data[1], parsed_values)
    
    return(ohlc_xts)
}

Example Use Case

To use this function, simply call it with no arguments.

# Call the transform_data function
transformed_data <- transform_data()
print(transformed_data)

Conclusion

In conclusion, transforming streaming data into an xts object containing Open, High, Low, and Close values can be achieved in R using a combination of regular expressions and the xts package. By breaking down the data format into its constituent parts, parsing these values, and creating an xts object from them, we have successfully transformed the streaming data.

This article has demonstrated how to create a Lightstreamer client in R, reproduce the streaming data visible in the console, transform this data into OHLC format using regular expressions and the xts package, and create an xts object containing these values.

Last modified on 2024-10-27