Skip to main content

Architecture Overview

This page provides a general overview of the OpenBB Platform architecture and the key Python classes most processes interact with.

Core Dependencies​

note

In reference to, "core", we mean the openbb-core package.

The Core relies on a set of carefully selected Python libraries, including:

  • FastAPI for building the API.
  • Uvicorn as the ASGI server.
  • Pandas for data manipulation and analysis.
  • Pydantic for data validation and serialization using Python type annotations.
  • Requests/AIOHTTP for making HTTP requests.
  • WebSockets for handling WebSocket connections.

The current specifications can be found in the pyproject.toml file of the GitHub repository.

Keeping A Lean Core​

Deploying Custom Applications

Keeping the application as lean as possible is crucial for maintaining its performance, and to eliminate bloat. Install only the components required to get the job(s) done.

Interface Overview​

With just the Core, there are no routers or endpoints. Users and developers are free to create their own combinations of extensions, tailored to suit specific requirements.

The openbb-core package comes with the ability to operate two, independent, interfaces.

  • Python Interface
    • Python application wrapping the installed routers.
    • A build script writes static files to create the application, which includes all installed extensions.
      • Docstrings and function signatures are generated by this process.
    • Configurable, and extendable, response object handling data validation, timestamping, and more.
  • REST API
    • An instance of FastAPI

Python Interface​

Key Information

The build script requires write access to the site-packages folder, post-installation, for generating static assets.

When a new extension is installed, or removed, the static assets must be regenerated in order to reflect the changes.

Remote deployments utilizing the interface should invoke the openbb-build command line script from the shell script establishing the environment, packages, and file system.

Build Script​

Using the import below initializes the application instance from static files generated by the build script.

from openbb import obb

The build script is not part of the openbb-core package. It is in the main package (pip install openbb), and it can be installed without any other components.

If openbb is not an installed package, but openbb-core is:

pip install openbb --no-deps

To build the assets:

import openbb

openbb.build()

Restart the Python interpreter to begin using the application with:

from openbb import obb

Alternatively, the process above can be run from the command line shell:

openbb-build

The bare interface includes:

  • /account
    • This module contains methods for logging in to, and updating, an OpenBB Hub account.
  • /user
  • /system
  • /coverage
    • Information and metadata about the data provider extensions and their definitions. providers
      • commands
      • command_model
      • command_schemas
      • reference
Python Settings

See system_settings.json for configuring runtime parameters.

API Interface​

Items described immediately above will be included as routers by setting the environment variable, OPENBB_DEV_MODE='1'

The FastAPI instance, with all installed routers and extensions, can be imported with:

from openbb_core.rest_api import app

Launched via Uvicorn:

uvicorn openbb_core.rest_api:app
API Settings

See system_settings.json for configuring runtime parameters.

Application Overview​

The application - both Python Interface and REST API - share core logic and models. It is the product of all installed extensions.

All routes are a subclass of the Router class.

Routers​

Router extensions are registered in the pyproject.toml file, and installed as a Python package.

[tool.poetry.plugins."openbb_core_extension"]
my_router = "my_router.my_router:router"

Router Class​

The Router class is a subclass of fastapi.APIRouter, helping validate and complete function signatures, ensuring that the functions registered as API endpoints have the correct parameters and return types.

Import​

from openbb_core.provider.app.router import Router

Initializing​

Initialize the class with a prefix and description.

some_router = Router(prefix="", description="Some description of this router.")
some_router.include_router(some_sub_router)

The api_router property provides access to the underlying APIRouter instance, allowing for direct interaction with the FastAPI routing system, if needed.

@command​

A @router.command decorator is exposed upon initialization, extending the functionality of FastAPI().add_api_route() to include additional parameters:

  • model
    • A model name shared by all Provider extensions feeding the same input - i.e, "EquityHistorical"
  • examples
    • Examples for use to include in documentation and Python docstring - see the page here for an explanation.
  • no_validate
    • A flag to disable response validation for the endpoint.
      • The output type is assigned as Any
  • All other arguments and keyword arguments are passed to APIRouter.add_api_route - i.e, openapi_extra=dict(some_key=some_value)

Response Model​

Define the response model as part of the function's definition.

@router.command(
    methods=["POST"],
    examples=[
        PythonEx(
            description="Calculate HMA with historical stock data.",
            code=[
                "stock_data = obb.equity.price.historical(symbol='TSLA', start_date='2023-01-01', provider='fmp')",
                "hma_data = obb.technical.hma(data=stock_data.results, target='close', length=50, offset=0)",
            ],
        ),
    ],
)
def hma(
    data: list[Data],
    target: str = "close",
    index: str = "date",
    length: int = 50,
    offset: int = 0,
) -> list[Data]: ...

Data Class​

The OpenBB Standardized Data Model, and is a subclass of pydantic.BaseModel.

This is the base class for all data returned via the ProviderInterface, and is used for POST requests to the data field.

Import​

from openbb_core.provider.abstract.data import Data

Key Features:

  • Dynamic field support: Dynamically handles fields that are not pre-defined in the model,
  • Alias handling: Maintain compatibility with different naming conventions across various data formats.
  • Datetime/string interoperability.

Example​

# Direct instantiation
data_record = Data(name="OpenBB", value=42)

# Conversion from a dictionary
data_dict = {"name": "OpenBB", "value": 42}
data_record = Data(**data_dict)

# __alias_dict__
class SomeData(Data):
    """Some Data Model."""

    __alias_dict__ = {
        "output_field": "someWhackyInputName",
    }
    output_field: str = Field(description="My Output")

SomeData(someWhackyInputName="Some Output")

Output:

SomeData(output_field=Some Output)

QueryParams Class​

The QueryParams class is a standardized model for handling query input parameters. Similar to Data, it is a subclass of pydantic.BaseModel.

Behaviour described here is mostly applicable to endpoints utilizing the ProviderInterface.

The class is used by the ProviderInterface to validate, merge, and discriminate endpoint parameters shared by multiple provider extensions.

Along with __alias_dict__, QueryParams uses __json_schema_extra__ to constrict parameter inputs, and to deterimine when more than one entry should be accepted as a list.

Import​

from openbb_core.provider.abstract.query_params import QueryParams

Example​

"""Some Query Model."""

from datetime import date as dateType
from typing import Optional
from warnings import warn

from openbb_core.provider.abstract.query_params import QueryParams
from pydantic import Field, field_validator


class SomeQueryParams(QueryParams):
    """Some Query Parameters."""

    __alias_dict__ = {
        "symbol": "ticker",
        "start_date": "begin",
        # "input_field": "output_alias"
    }
    __json_schema_extra__ = {
        "symbol": {"multiple_items_allowed": True, "choices": SOME_SYMBOL_LIST},
        "interval": {"multiple_items_allowed": False},  # No need to define this, it is the default behaviour. 
    }

    symbol: str = Field(description="The ticker symbol.")
    interval: Literal["1d", "1w", "1m"] = Field(
        default="1d",
        description="The interval of the timeseries. Choices: ["1d", "1w", "1m"]",
    )
    start_date: Optional[dateType] = Field(
        default=None,
        description="The start date, as YYYY-MM-SS, or datetime.date object",
    )
    end_date: Optional[dateType] = Field(
        default=None,
        description="The end date, as YYYY-MM-SS, or datetime.date object",
    )

    @field_validator("symbol", mode="before", check_fields=False)
    @classmethod
    def _validate_symbol(cls, v) -> str:
        """Validate the symbol."""
        if not v:
            raise ValueError("Please enter a symbol.")
        symbol = v if isinstance(v, list) else v.upper().split(",")

        new_symbols: list = []

        for s in symbol:
            if s not in SOME_SYMBOL_LIST:
                msg = f"Invalid symbol provided {s}"
                warn(msg)
                continue
            new_symbols.append(s)

        if not new_symbols:
            raise ValueError(f"No valid symbols supplied. Choices are: {SOME_SYMBOL_LIST}")

        return ",".join(new_symbols)

ProviderInterface​

The ProviderInterface is the map of all installed provider extensions to their respective callables, and is a Singleton accepting no initialization parameters.

It is responsible for handling the provider parameter - i.e, provider="yfinance" at execution, and is called by internal processes.

There is not a general need to interact with this class directly; however, it is useful to know that it exists.

Import​

from openbb_core.app.provider_interface import ProviderInterface

Each item in the ProviderInterface maps to a Fetcher, which executes the TET pattern.

Provider Extensions​

A provider extension is created by defining pyproject.toml and installing the package.

[tool.poetry.plugins."openbb_provider_extension"]
some_provider = "some_provider:some_provider"

It is a collection of data-collecting Fetchers that should be added to the application via the provider parameter.

Provider Class​

The Provider class is initialized in the __init__.py file of a provider extension module.

Import​

from openbb_core.provider.abstract.provider import Provider

It maps the Router to the Provider, and defines other key metadata for the extension. The example below is from openbb-bls.

This is how the ProviderInterface maps any @router.command(model="SomeModel") function to all known instances.

Example​

"""BLS Provider Module."""

from openbb_bls.models.search import BlsSearchFetcher
from openbb_bls.models.series import BlsSeriesFetcher
from openbb_core.provider.abstract.provider import Provider

bls_provider = Provider(
    name="bls",
    website="https://www.bls.gov/developers/api_signature_v2.htm",
    description="The Bureau of Labor Statistics' (BLS) Public Data Application Programming Interface (API)"
    + " gives the public access to economic data from all BLS programs."
    + " It is the Bureau's hope that talented developers and programmers will use the BLS Public Data API to create"
    + " original, inventive applications with published BLS data.",
    credentials=["api_key"],
    fetcher_dict={
        "BlsSearch": BlsSearchFetcher,
        "BlsSeries": BlsSeriesFetcher,
    },
    repr_name="Bureau of Labor Statistics' (BLS) Public Data API",
    instructions="Sign up for a free API key here: https://data.bls.gov/registrationEngine/",
)

The TET Pattern​

Each Fetcher follows this pattern. It stands for Transform, Extract, Transform.

The workflow divides data collection into three, seperate, tasks.

Diagram

Fetcher Class​

info

The Fetcher class imposes a standardized structure, namely:

  • Transform the query: output of this method should be QueryParams child.
  • Extract the data: output of this method can be Any but it's recommended to be a dict.
  • Transform the data: output of this method should be a List[Data] or Data (or a child of it - i.e. SomeData).
  • Built-in test for verifying basic operation at each stage.

Import​

from openbb_core.provider.abstract.fetcher import Fetcher

Methods​

The generic methods are meant to be overwritten with the implementation.


class SomeFetcher(Fetcher[SomeQueryParams, list[SomeData]]):
    """Some Fetcher."""

    # Tell query executor if credentials are required. Can be overridden by subclasses.
    # Useful if a provider has some endpoints requiring API keys, but not all.
    require_credentials = False

    @staticmethod
    def transform_query(params: dict[str, Any]) -> SomeQueryParams:
        """Transform the params to the provider-specific query."""
        raise NotImplementedError

    @staticmethod
    async def aextract_data(query: SomeQueryParams, credentials: Optional[dict[str, str]], **kwargs: Any) -> dict:
        """Asynchronously extract the data from the provider."""
        raise NotImplementedError

    @staticmethod
    def extract_data(query: SomeQueryParams, credentials: Optional[dict[str, str]], **kwargs: Any) -> dict:
        """Extract the data from the provider."""
        raise NotImplementedError

    @staticmethod
    def transform_data(query: SomeQueryParams, data: dict, **kwargs: Any) -> list[SomeData]:
        """Transform the provider-specific data."""
        raise NotImplementedError
Notes
  • One of, extract_data, or, aextract_data, must be implemented.
  • All parameters for these methods are positional and should be defined as described above.

Execution Method​

A Fetcher can be executed without initialization, as an async function. It requires two positional arguments as dictionaries:

  • query - a dictionary to be converted into the QueryParams class.
  • credentials - a dictionary of credentials required for the provider and endpoint.
    • Pass as an empty dictionary when no credentials are required.
results = await SomeFetcher.fetch_data({"symbol": "btcusd"}, {})

The return is the output of SomeFetcher.tranform_data

Router Definition​

To implement the ProviderInterface as a Router endpoint, follow the pattern below in the extension's router file.


from openbb_core.app.model.command_context import CommandContext  # A FastAPI Depends injection with the initialized user and system preferences
from openbb_core.app.model.obbject import OBBject  # The OpenBB standard response object with output validation.
from openbb_core.app.provider_interface import (
    ExtraParams, # All provider-specific parameters.
    ProviderChoices, # All providers feeding the endpoint.
    StandardParams,  # Parameters defined in the common 'Standard' model, if any.
)
from openbb_core.app.query import Query  # The query executor.
from openbb_core.app.router import Router


router = Router(prefix="", description="Some Router.")


@router.command(
    model="SomeModel",
    examples=[
        APIEx(parameters={"symbol": "btcusd", "interval": "1d", "provider": "some_provider"}),
    ],
)
async def some_function(
    cc: CommandContext,
    provider_choices: ProviderChoices,
    standard_params: StandardParams,
    extra_params: ExtraParams,
) -> OBBject:
    """Use the `SomeModel` Fetcher."""
    return await OBBject.from_query(Query(**locals()))

OBBject Class​

The OBBject class is the standard response object from output functions, and has the following attributes:

  • results
    • Serializable data that was returned by the command.
  • provider
    • The name of the provider that was used to obtain the data, if any.
  • warnings
    • Warnings generated by the command execution.
  • extra
    • Additional metadata about the command run, including any arguments, the route, the timestamp, etc.

It is extendable, similar to a Pandas extension, and has methods for converting the results to different formats.

Import​

from openbb_core.app.model.obbject import OBBject

A bare instance can be created by defining results as None.

obbject = OBBject(results=None)

Output Conversion​

Various formats and filtering can be applied to the function's output.

model_dump()​

Pydantic method for converting the complete object to a Python dictionary.

model_dump_json()​

Pydantic method for converting the complete object to a serialized JSON string.

to_df()​

Alias for to_dataframe(), and converts the contents of obbject.results to a Pandas DataFrame.

to_dict()​

Converts the contents of obbject.results to a Python dictionary object with a given orientation.

to_numpy()​

Convert the contents of obbject.results to a NumPy array.

to_polars()​

Converts the contents of obbject.results to a Polars DataFrame.

note

Polars is not included with OpenBB packages and must be installed.

OBBject Extensions​

An OBBject extension is created by defining pyproject.toml and installing the package.

[tool.poetry.plugins."openbb_obbject_extension"]
some_extension = "some_extension:ext"

The module's __init__.py file creates an instance of the extension object, and is attached to the output of every command.

from openbb_core.app.model.extension import Extension

ext = Extension(name="some_extension")

@ext.obbject_accessor
class SomeExtension:
    def __init__(self, obbject):
        self._obbject = obbject

    def hello(self):
        print(f"Say Hello!")

An example of this type of extension is, openbb-charting.

On this page