Skip to main content

Architecture Overview

This page provides a general overview of the OpenBB Platform architecture and the key Python classes most processes interact with.

Core Dependencies​

note

In reference to, "core", we mean the openbb-core package.

The OpenBB Platform core relies on a set of carefully selected Python libraries. These include:

  • FastAPI for building the API.
  • Uvicorn as the ASGI server.
  • Pandas for data manipulation and analysis.
  • Pydantic for data validation and serialization using Python type annotations.
  • Requests/AIOHTTP for making HTTP requests.
  • WebSockets for handling WebSocket connections.

These dependencies are specified in the pyproject.toml files.

Importance Of A Lean Core​

Keeping the OpenBB Platform core as lean as possible is crucial for maintaining the its performance, ease of use, and flexibility. A lean core means faster installation times, less memory usage, and overall better performance. It also reduces the risk of conflicts between dependencies and makes the platform easier to maintain and update.

Moreover, a lean core allows for greater flexibility. Users of the platform can add additional functionality through extensions without being burdened by unnecessary core dependencies. This makes the OpenBB Platform adaptable to a wide range of use cases and requirements.

Python Interface​

When using the OpenBB Platform from a Python Interface, docstrings and type hints are your best friends as it provides plenty of context on how to use the commands.

For example, obb.technical.ema?:

obb.technical.ema(
    data: Union[list, dict, pandas.core.frame.DataFrame, List[pandas.core.frame.DataFrame], pandas.core.series.Series, List[pandas.core.series.Series], numpy.ndarray, openbb_core.provider.abstract.data.Data, List[openbb_core.provider.abstract.data.Data]],
    target: str = 'close',
    index: str = 'date',
    length: int = 50,
    offset: int = 0,
    chart: bool = False,
) -> openbb_core.app.model.obbject.OBBject

We can easily deduct that the ema command accepts data in a number of formats.

Other types might be added in the future.

API Interface​

When using the OpenBB Platform via an API Interface, the types are a bit more constrained than the Python interface. For example, we can't use pandas.DataFrame as a type. However the same principles apply for what Data means - i.e., any data processing command, which are characterized as POST endpoints on the API, will accept data as a list of records, in the request body

[
    {
        "open": 80,
        "high": 80.69,
        "low": 77.37,
        "close": 77.62,
        "volume": 2487300
    }
]

QueryParams Class​

The QueryParams class is a standardized model for handling query input parameters in the OpenBB platform. It extends the BaseModel from the Pydantic library, which provides runtime data validation and serialization.

Below is the EquityHistorical standard model.

"""Equity Historical Price Standard Model."""

from datetime import (
    date as dateType,
    datetime,
)
from typing import Optional, Union

from dateutil import parser
from pydantic import Field, field_validator

from openbb_core.provider.abstract.data import Data
from openbb_core.provider.abstract.query_params import QueryParams
from openbb_core.provider.utils.descriptions import (
    DATA_DESCRIPTIONS,
    QUERY_DESCRIPTIONS,
)

class EquityHistoricalQueryParams(QueryParams):
    """Equity Historical Price Query."""

    symbol: str = Field(description=QUERY_DESCRIPTIONS.get("symbol", ""))
    interval: Optional[str] = Field(
        default="1d",
        description=QUERY_DESCRIPTIONS.get("interval", ""),
    )
    start_date: Optional[dateType] = Field(
        default=None,
        description=QUERY_DESCRIPTIONS.get("start_date", ""),
    )
    end_date: Optional[dateType] = Field(
        default=None,
        description=QUERY_DESCRIPTIONS.get("end_date", ""),
    )

    @field_validator("symbol", mode="before", check_fields=False)
    @classmethod
    def to_upper(cls, v: str) -> str:
        """Convert field to uppercase."""
        return v.upper()
info

Note that not all possible parameters are defined here, and can be further refined in the provider-specific model.

For example, interval is defined as a string in the standard model, but is defined again as a Literal with explicit choices specific to that source. Here, it would not be possible to define all valid choices in a way that is compatible with all providers.

The class can have a dictionary, __alias_dict__, which can be used to map the original parameter names to aliases. This can be useful when dealing with different data providers that may use different naming conventions for similar parameters.

The __json_schema_extra__ dictionary can be used to define whether multiple items are accepted by a parameter.

Additionally, "choices", is a list of accepted vales for the parameter. Adding them here ensures they are included in openapi.json.

__json_schema_extra__ = {
    "symbol": {"multiple_items_allowed": True, "choices": SOME_SYMBOL_LIST},
    "category": {"multiple_items_allowed": False},  # No need to define this, it is the default behaviour. 
}

The __repr__ method provides a string representation of the QueryParams object, which includes the class name and a list of the model's parameters and their values.

The model_config attribute is a ConfigDict instance that allows extra fields not defined in the model and populates the model by name.

The model_dump method is used to serialize the model into a dictionary. If the __alias_dict__ is not empty, it will use the aliases defined in it for the keys in the returned dictionary. If the __alias_dict__ is empty, it will return the original serialized model.

Data Class​

The OpenBB Standardized Data Model.

The Data class is a flexible Pydantic model designed to accommodate various data structures for OpenBB's data processing pipeline as it's structured to support dynamic field definitions.

The model leverages Pydantic's powerful validation features to ensure data integrity while providing the flexibility to handle extra fields that are not explicitly defined in the model's schema.

This makes the Data class ideal for working with datasets that may have varying structures or come from heterogeneous sources.

Key Features:

  • Dynamic field support: Can dynamically handle fields that are not pre-defined in the model, allowing for great flexibility in dealing with different data shapes.
  • Alias handling: Utilizes an aliasing mechanism to maintain compatibility with different naming conventions across various data formats.

Usage:

The Data class can be instantiated with keyword arguments corresponding to the fields of the expected data. It can also parse and validate data from JSON or other serializable formats, and convert them to a Data instance for easy manipulation and access.

Example:

from openbb_core.provider.abstract.data import Data

# Direct instantiation
data_record = Data(name="OpenBB", value=42)

# Conversion from a dictionary
data_dict = {"name": "OpenBB", "value": 42}
data_record = Data(**data_dict)

The class is highly extensible and can be subclassed to create more specific models tailored to particular datasets or domains, while still benefiting from the base functionality provided by the Data class.

The Data model for the EquityHistorical standard model is shown below.

class EquityHistoricalData(Data):
    """Equity Historical Price Data."""

    date: Union[dateType, datetime] = Field(
        description=DATA_DESCRIPTIONS.get("date", "")
    )
    open: float = Field(description=DATA_DESCRIPTIONS.get("open", ""))
    high: float = Field(description=DATA_DESCRIPTIONS.get("high", ""))
    low: float = Field(description=DATA_DESCRIPTIONS.get("low", ""))
    close: float = Field(description=DATA_DESCRIPTIONS.get("close", ""))
    volume: Optional[Union[float, int]] = Field(
        default=None, description=DATA_DESCRIPTIONS.get("volume", "")
    )
    vwap: Optional[float] = Field(
        default=None, description=DATA_DESCRIPTIONS.get("vwap", "")
    )

    @field_validator("date", mode="before", check_fields=False)
    def date_validate(cls, v):
        """Return formatted datetime."""
        if ":" in str(v):
            return parser.isoparse(str(v))
        return parser.parse(str(v)).date()

Fetcher Class​

The Fetcher class is an abstract base class designed to provide a structured way to fetch data from various providers. It uses generics to allow for flexibility in the types of queries, data, and return values it handles.

The class defines a series of methods that must be implemented by any subclass: transform_query, extract_data, and transform_data. These methods represent the core steps of fetching data: transforming input parameters into a provider-specific query, extracting data from the provider using the query, and then transforming the provider-specific data into a desired format.

The fetch_data method orchestrates these steps, taking in parameters and optional credentials, and returning the transformed data.

The class also includes a test method for validating the functionality of a fetcher, performing assertions on each stage of the fetch process.

Additionally, the Fetcher class uses a custom classproperty decorator to define class-level properties that return the types of the query parameters, return value, and data.

The require_credentials class variable indicates whether credentials are needed to fetch data from the provider. This can be overridden by subclasses as needed.

info

The Fetcher class implementation is based on the TET pattern. This pattern imposes a standardized structure, namely:

  • Transform the query: the output of this method should be QueryParams child.
  • Extract the data: the output of this method can be Any but it's recommended to be a Dict (will facilitate the next step of the fetcher's action, the data transformation).
  • Transform the data: the output of this method should be a List[Data] or Data (or a child of it).

OBBject Class​

The OBBject class is a generic class in the OpenBB platform that represents a standardized object for handling and manipulating data fetched from various providers. It extends the Tagged class and uses Python's generics to allow flexibility in the type of results it can handle.

The class includes several fields such as results, provider, warnings, chart, and extra, which respectively represent the fetched data, the data provider, any warnings generated during data fetching, an optional chart object for visualizing the data, and a dictionary for any additional information.

The class provides several methods for converting the fetched data into different formats, including to_df (or to_dataframe) for converting to a pandas DataFrame, to_polars for converting to a Polars DataFrame, to_numpy for converting to a numpy array, and to_dict for converting to a dictionary.

The class also includes a __repr__ method for a human-readable representation of the object, as well as the familiar Pydantic BaseModel methods like model_dump() and model_json_schema().

Router Class​

The Router class in the OpenBB platform is responsible for managing and routing API requests. It uses the APIRouter from the FastAPI library to handle routing.

The class includes a command method that allows for the registration of callable functions as API endpoints. This method takes care of setting up the API route, including defining the HTTP methods, response models, operation IDs, and other necessary parameters for the API endpoint.

The include_router method allows for the inclusion of another router, effectively merging the routes from the included router into the current one.

The Router class also interacts with the SignatureInspector class to validate and complete function signatures, ensuring that the functions registered as API endpoints have the correct parameters and return types.

The api_router property provides access to the underlying APIRouter instance, allowing for direct interaction with the FastAPI routing system if needed.

info

The Router class exposes a command decorator that allows for the registration of callable functions as API endpoints. This method takes care of setting up the API route, including defining the HTTP methods, response models, operation IDs, and other necessary parameters for the API endpoint.

This decorator enforces that the decorated function (of type Callable) returns an OBBject instance. I.e., the signature of a decorated function should translate to Callable[P, OBBject] (a callable object (like a function) that takes arguments of type P and returns an object of type OBBject).

Import Statements​


# The `Data` class
from openbb_core.provider.abstract.data import Data

# The `QueryParams` class
from openbb_core.provider.abstract.query_params import QueryParams

# The `Fetcher` class
from openbb_core.provider.abstract.fetcher import Fetcher

# The `OBBject` class
from openbb_core.app.model.obbject import OBBject

# The `Router` class
from openbb_core.app.router import Router

The TET Pattern​

The TET pattern is a pattern that we use to build the Fetcher classes. It stands for Transform, Extract, Transform.

Diagram

As the OpenBB Platform has its own standardization framework and the data fetcher are a very important part of it, we need to ensure that the data is transformed and extracted in a consistent way, to help us do that, we came up with the TET pattern, which helps us build and ship faster as we have a clear structure on how to build the Fetcher classes.

  1. Transform query

    transform_query(params: Dict[str, Any])

    Transforms the query parameters. Given a params dictionary this method should return the transformed query parameters as a QueryParams child so that we can leverage the pydantic model schemas and validation into the next step. This might also be the place do perform some transformations on any given parameter, i.e., if you want to transform an empty date into a datetime.now().date().

  2. Extract data

    extract_data(query: ExampleQueryParams,credentials: Optional[Dict[str, str]],**kwargs: Any,) -> Dict

    Makes the request to the API endpoint and returns the raw data. Given the transformed query parameters, the credentials and any other extra arguments, this method should return the raw data as a dictionary.

  3. Transform data

    transform_data(query: ExampleQueryParams, data: Dict, **kwargs: Any) -> List[ExampleHistoricalData]

    Transforms the raw data into the defined data model. Given the transformed query parameters (might be useful for some filtering), the raw data and any other extra arguments, this method should return the transformed data as a list of Data children.

Data Processing Commands​

The data processing commands are commands that are used to process the data that may or may not come from the OpenBB Platform.

In order to create a data processing framework general enough to be used by any extension, we've created a special abstract class called Data which all standardized (and consequently its child classes) will inherit from.

Why is this important?

We ensure that all OBBject.results will share a common ground on which we can apply out-of-the-box data processing commands, such as the ta, qa, or the econometrics menus.

But what's really the Data class?

It's a pydantic model that inherits from the BaseModel and can contain any given number of extra fields. In practice, it looks as follows:

from openbb import obb

res = obb.equity.price.historical("AAPL")
res.results[0]
AVEquityHistoricalData(date=2023-11-03 00:00:00, open=174.24, high=176.82, low=173.35, close=176.65, volume=79829246.0, vwap=None, adj_close=None, dividend_amount=None, split_coefficient=None)

The AVEquityHistoricalData class, is a child class of the Data class.

Note how we've indexed to get only the first element of the results list (which represents a single row, if we want to think about it as a tabular output).

This simply means that we are getting a List of AVEquityHistoricalData from the obb.equity.price.historical command. Or, we can also say that that's equivalent to List[Data]!

This is very powerful, as we can now apply any data processing command to the results list, without worrying about the underlying data structure.

That's why, on data processing commands (such as the ta menu) we will find on its function signature the following:


def ema(
        self,
        data: Union[List[Data], pandas.DataFrame],
        target: str = "close",
        index: str = "date",
        length: int = 50,
        offset: int = 0,
        chart: bool = False,
    ) -> OBBject[List[Data]]:

    ...

Note that data can actually be a different type, but we'll focus on the List[Data] case for now.

Does that mean that I can only use the data processing commands if I instantiate a class that inherits from Data? Not at all! Consider the following example:


from openbb_core.provider.abstract.data import Data
my_data_item_1 = {"open": 1, "high": 2, "low": 3, "close": 4, "volume": 5, "date": "2020-01-01"}
my_data_item_1_as_data = Data.model_validate(my_data_item_1)
my_data_item_1_as_data
Data(open=1, high=2, low=3, close=4, volume=5, date="2020-01-01")

This means that the Data class is clever enough to understand that you are passing a dictionary and it will try to validate it for you.

In other words, if you're using data that doesn't come from the OpenBB Platform, you only need to ensure it's parsable by the Data class and you'll be able to use the data processing commands.

In other words, imagine you have a dataframe that you want to use with the ta menu. You can do the following:


res = obb.equity.price.historical("AAPL")
my_df = res.to_dataframe() # yes, you can convert your OBBject.results into a dataframe out-of-the-box!
my_records = df.to_dict(orient="records")

obb.ta.ema(data=my_record)
OBBject

results: [{'close': 77.62, 'close_EMA_50': None}, {'close': 80.25, 'close_EMA_50': ...}] # this is a `List[Data]` yet again

Note that that for this example we've used the OBBject.to_dataframe() method to have an example dataframe, but it could be any other dataframe that you have.

On this page