Fetch Data From a New Provider

This page demonstrates how to get started mapping Provider extension models to the Provider Interface and Router.

It continues the example provided here. By the end, you will have mapped a new Provider extension to the World News standard model. This will connect it to the existing command and make it accessible through the provider parameter.

All Provider models are made of three elements:

• QueryParams
• Data
• Fetcher

Start Mapping

Create Model File

• In the /models folder of the extension - /Users/username/path_to_created_folder/empty_provider/openbb_empty_provider/models - create a new file called, "world_news.py".
• Start the file with a docstring, then a new line.

"""Empty World News Model."""

Import Statements

The import schema will follow a pattern similar to this, depending on the specific requirements of the function.

from typing import Any, Dict, List, Literal, Optional

from openbb_core.provider.abstract.fetcher import Fetcher
from openbb_core.provider.standard_models.world_news import WorldNewsQueryParams, WorldNewsData

from pydantic import Field

Create a QueryParams Model

• Create a class that inherits from the standard model.

class EmptyWorldNewsQueryParams(WorldNewsQueryParams):
    """Empty World News Query Params."""

    some_param: Optional[str] = Field(
        default=None,
        description="Some Empty Parameter.",
    )

Field and model validators are added here, if required.

Create a Data Model

• Create a class that inherits from the standard model.

class EmptyWorldNewsData(WorldNewsData):
    """Empty World News Data"""

    some_param: Optional[str] = Field(
        default=None,
        description="Some Empty Data Field."
    )

Field and model validators are added here, if required.

Build Fetcher

The Fetcher consists of three stages:

• Transform Query
• Extract Data
• Transform Data

Each stage is a static method within the Fetcher class that inherits the QueryParams and Data models.

The structure of the methods within should always be the same, and input/output types should match throughout.

class EmptyWorldNewsFetcher(
    Fetcher[
        EmptyWorldNewsQueryParams,
        List[EmptyWorldNewsData],
    ]
):
    """Empty World News Fetcher."""

    @staticmethod
    def transform_query(params: Dict[str, Any]) -> EmptyWorldNewsQueryParams:
        """Transform query params."""
        new_params = params
        if params.get("some_param"):
            new_params["some_param"] = "do_something"
        return EmptyWorldNewsQueryParams(**new_params)

    @staticmethod
    async def aextract_data( # The function does not have to be asynchronous. If not, remove the leading 'a', 'extract_data'.
        query: EmptyWorldNewsQueryParams,
        credentials: Optional[Dict[str, str]],
        **kwargs: Any,
    ) -> List[Dict]:
        """Extract data. Put HTTP Requests here. This should return the closest form of raw data possible."""
        results = {
            "date": datetime.now().date(),
            "title": "Hello from the Empty Provider extension!",
            "some_param": query.some_param,
        }
        return [results]

    @staticmethod
    def transform_data(
        query: EmptyWorldNewsQueryParams,
        data: List[Dict],
        **kwargs: Any,
    ) -> List[EmptyWorldNewsData]:
        """Transform data. Put parsing logic here, if required. This should return validated data models."""
        return [EmptyWorldNewsData.model_validate(d) for d in data]

2. Map Model To Provider Interface

• In the __init__.py where the Provider class was defined, import the Fetcher from the model file and map it to the router.

# /Users/username/path_to_created_folder/empty_provider/openbb_empty_provider/__init__.py
"""Empty Provider Module."""

from openbb_empty_provider.models.world_news import EmptyWorldNewsFetcher
from openbb_core.provider.abstract.provider import Provider

empty_provider = Provider(
    name="empty",
    website="http://empty.io",
    description="""The empty provider is a supplier of promises.""",
    # credentials=["api_key"],
    fetcher_dict={
        "WorldNews": EmptyWorldNewsFetcher
    },
)

3. Rebuild Static Assets

• The Python Interface and static assets need to be rebuilt after these changes.

python -c "import openbb; openbb.build()"

4. Smoke Test

The function's docstring should list the new provider, and the command should return an OBBject instance.

    Parameters
    ----------
    limit : int
        The number of data entries to return. The number of articles to return.
    start_date : Union[datetime.date, None, str]
        Start date of the data, in YYYY-MM-DD format.
    end_date : Union[datetime.date, None, str]
        End date of the data, in YYYY-MM-DD format.
    provider : Optional[Literal['benzinga', 'empty', 'fmp', 'intrinio', 'tiingo'...
        The provider to use for the query, by default None.

from openbb import obb
obb.news.world(provider="empty")

OBBject

id: 066356fc-9661-7448-8000-5885e5120efb
results: [{'date': datetime.datetime(2024, 5, 3, 0, 0), 'title': 'Hello from the Em...
provider: empty
warnings: None
chart: None
extra: {'metadata': {'arguments': {'provider_choices': {'provider': 'empty'}, 'stan...

The parameters supplied are captured under results.extra["metadata"].arguments.

response = obb.news.world(provider="empty", some_param="adjustment")
response.extra["metadata"].arguments

{
    'provider_choices': {'provider': 'empty'}, 'standard_params': {'limit': 2500, 'start_date': None, 'end_date': None},
    'extra_params': {'some_param': 'adjustment'}
}

The transformed parameters are passed through to the function.

response.results

>>> response.results
[EmptyWorldNewsData(date=2024-05-03 00:00:00, title=Hello from the Empty Provider extension!, images=None, text=None, url=None, some_param=do_something)]

Conclusion

This process created, built, and mapped a new OpenBB Platform Provider model to an existing router endpoint and standard model.

The output was validated and returned as an instance of OBBject.

Create a new model for each endpoint needing to be mapped.