Value-based search¶

A value-based search finds the intervals in time where one or more tags meet a condition — for example, every period where a flow rate stays above a threshold. This notebook builds one up in three steps:

1. a basic search,
2. the same search enriched with calculations, and
3. a second search run within the results of the first.

Each step returns a PagedDataFrame; we call .collect() to materialise it into a pandas DataFrame whose IntervalIndex holds the found events.

Setup¶

We start by authenticating a client and retrieving the tags we need for our example.

In [1]:

import os

import pandas as pd
from dotenv import find_dotenv, load_dotenv

from trendminer_interface import client_from_credentials

# Credentials and server details are securely stored in .env.docs for our example code.
# find_dotenv walks up from the working directory to locate it, so this works
# regardless of the directory the notebook is run from.
load_dotenv(find_dotenv(".env.docs", usecwd=True))

client = client_from_credentials(
    url=os.environ["TM_URL"],
    client_id=os.environ["TM_CLIENT_ID"],
    client_secret=os.environ["TM_CLIENT_SECRET"],
)

level = client.tag.from_name("TM6-BP2-LEVEL.1")
temp = client.tag.from_name("TM6-BP2-TEMP.1")
product = client.tag.from_name("TM6-BP2-PRODUCT.1")

import os import pandas as pd from dotenv import find_dotenv, load_dotenv from trendminer_interface import client_from_credentials # Credentials and server details are securely stored in .env.docs for our example code. # find_dotenv walks up from the working directory to locate it, so this works # regardless of the directory the notebook is run from. load_dotenv(find_dotenv(".env.docs", usecwd=True)) client = client_from_credentials( url=os.environ["TM_URL"], client_id=os.environ["TM_CLIENT_ID"], client_secret=os.environ["TM_CLIENT_SECRET"], ) level = client.tag.from_name("TM6-BP2-LEVEL.1") temp = client.tag.from_name("TM6-BP2-TEMP.1") product = client.tag.from_name("TM6-BP2-PRODUCT.1")

A basic search¶

A value-based search is built from one or more (tag, operator, value) queries and a minimum duration, then run over a time window with get_results. The result is a PagedDataFrame whose IntervalIndex holds the events where the condition held — here, every period of at least 15 minutes where the level stays above 10. The open column tells us whether a result is still ongoing at the end of the search window.

In [2]:

# Time window to search in
search_window = pd.Interval(
    pd.Timestamp("2025-01-01", tz="UTC"),
    pd.Timestamp("2026-01-01", tz="UTC"),
)

high_level = client.search.value.define(
    queries=[
        (level, ">", 10),
    ],
    duration=pd.Timedelta(minutes=15),
)

df_high_level = high_level.get_results(target=search_window).collect()
df_high_level

# Time window to search in search_window = pd.Interval( pd.Timestamp("2025-01-01", tz="UTC"), pd.Timestamp("2026-01-01", tz="UTC"), ) high_level = client.search.value.define( queries=[ (level, ">", 10), ], duration=pd.Timedelta(minutes=15), ) df_high_level = high_level.get_results(target=search_window).collect() df_high_level

Out[2]:

	open
(2025-01-01 00:27:00+00:00, 2025-01-01 01:22:00+00:00]	False
(2025-01-01 01:48:00+00:00, 2025-01-01 02:48:00+00:00]	False
(2025-01-01 03:09:00+00:00, 2025-01-01 04:09:00+00:00]	False
(2025-01-01 04:31:00+00:00, 2025-01-01 05:27:00+00:00]	False
(2025-01-01 05:52:00+00:00, 2025-01-01 06:46:00+00:00]	False
...	...
(2025-02-11 04:39:00+00:00, 2025-02-11 05:38:00+00:00]	False
(2025-02-11 06:00:00+00:00, 2025-02-11 07:01:00+00:00]	False
(2025-02-11 07:21:00+00:00, 2025-02-11 08:19:00+00:00]	False
(2025-02-11 08:42:00+00:00, 2025-02-11 09:39:00+00:00]	False
(2025-02-11 10:03:00+00:00, 2025-02-11 11:03:00+00:00]	False

737 rows × 1 columns

Adding calculations¶

By default a search returns only the event intervals. Passing calculations attaches computed columns to each event — each is a (tag, method, unit) tuple, keyed by the resulting column name. Here we record the product grade at the start of each event and the mean level during it.

In [3]:

high_level = client.search.value.define(
    queries=[
        (level, ">", 10),
    ],
    duration=pd.Timedelta(minutes=15),
    calculations={
        "product": (product, "start", ""),
        "mean_level": (level, "mean", "m"),
    },
)

df_high_level = high_level.get_results(target=search_window).collect()
df_high_level

high_level = client.search.value.define( queries=[ (level, ">", 10), ], duration=pd.Timedelta(minutes=15), calculations={ "product": (product, "start", ""), "mean_level": (level, "mean", "m"), }, ) df_high_level = high_level.get_results(target=search_window).collect() df_high_level

Out[3]:

	open	product	mean_level
(2025-01-01 00:27:00+00:00, 2025-01-01 01:22:00+00:00]	False	ALPHA	29.764614
(2025-01-01 01:48:00+00:00, 2025-01-01 02:48:00+00:00]	False	ALPHA	30.970741
(2025-01-01 03:09:00+00:00, 2025-01-01 04:09:00+00:00]	False	BETA	31.090126
(2025-01-01 04:31:00+00:00, 2025-01-01 05:27:00+00:00]	False	BETA	27.405737
(2025-01-01 05:52:00+00:00, 2025-01-01 06:46:00+00:00]	False	BETA	27.991938
...	...	...	...
(2025-02-11 04:39:00+00:00, 2025-02-11 05:38:00+00:00]	False	KAPPA	30.995026
(2025-02-11 06:00:00+00:00, 2025-02-11 07:01:00+00:00]	False	KAPPA	30.814535
(2025-02-11 07:21:00+00:00, 2025-02-11 08:19:00+00:00]	False	KAPPA	31.318115
(2025-02-11 08:42:00+00:00, 2025-02-11 09:39:00+00:00]	False	KAPPA	29.384530
(2025-02-11 10:03:00+00:00, 2025-02-11 11:03:00+00:00]	False	ALPHA	29.902628

737 rows × 3 columns

Searching within results¶

We can search directly within other events (a DataFrame with IntervalIndex). The existing columns (calculations) of the target will be kept, showing for each new search result the values of the original interval in which the new result sits.

In [4]:

high_temp = client.search.value.define(
    queries=[
        (temp, ">", 20),
    ],
    duration=pd.Timedelta(minutes=2),
    calculations={
        "max_temp": (temp, "max", "degC"),
    }
)

df_high_temp = high_temp.get_results(target=df_high_level).collect()
df_high_temp

high_temp = client.search.value.define( queries=[ (temp, ">", 20), ], duration=pd.Timedelta(minutes=2), calculations={ "max_temp": (temp, "max", "degC"), } ) df_high_temp = high_temp.get_results(target=df_high_level).collect() df_high_temp

Out[4]:

	open	max_temp	open	product	mean_level
(2025-01-01 07:55:00+00:00, 2025-01-01 07:56:00+00:00]	False	21.143122	False	GAMMA	31.593586
(2025-01-01 09:16:00+00:00, 2025-01-01 09:17:00+00:00]	False	20.207884	False	GAMMA	29.776937
(2025-01-01 14:40:00+00:00, 2025-01-01 14:41:00+00:00]	False	20.220330	False	GAMMA	30.806322
(2025-01-01 17:22:00+00:00, 2025-01-01 17:23:00+00:00]	False	20.794323	False	KAPPA	30.445500
(2025-01-01 18:43:00+00:00, 2025-01-01 18:44:00+00:00]	False	21.034365	False	KAPPA	29.513720
...	...	...	...	...	...
(2025-02-09 15:34:00+00:00, 2025-02-09 15:35:00+00:00]	False	20.620161	False	BETA	29.523663
(2025-02-09 16:55:00+00:00, 2025-02-09 16:56:00+00:00]	False	20.776815	False	BETA	30.944952
(2025-02-10 03:43:00+00:00, 2025-02-10 03:44:00+00:00]	False	20.361876	False	DELTA	30.100931
(2025-02-11 01:19:00+00:00, 2025-02-11 01:20:00+00:00]	False	21.148283	False	GAMMA	30.341885
(2025-02-11 08:04:00+00:00, 2025-02-11 08:05:00+00:00]	False	20.378036	False	KAPPA	31.318115

181 rows × 5 columns