feat(bigtable): client side metrics handlers by daniel-sanche · Pull Request #16760 · googleapis/google-cloud-python

daniel-sanche · 2026-04-22T00:08:21Z

Migrate googleapis/python-bigtable#1189 to the monorepo

This PR builds off of googleapis/python-bigtable#1187 to add handlers to the client-side metrics system, which can subscribe to the metrics stream, and export the results into different collection systems

We add two handlers to the system:

GoogleCloudMetricsHandler: sends metrics to a private OpenTelemetry meter, and then periodically exports them to GCP. Built on top of OpenTelemetryMetricsHandler
OpenTelemetryMetricsHandler: sends metrics to the root MeterProvider, so the user can access the exported metrics for their own systems. This will be off by default, but can be added alongside GoogleCloudMetricsHandler if needed

…_2_instrumentation_advanced

gemini-code-assist

Code Review

This pull request implements client-side metrics for the Bigtable library using OpenTelemetry, including a custom exporter for Google Cloud Monitoring. The review feedback focuses on improving resource efficiency by moving the MeterProvider and client_uid generation to the client level to avoid thread leaks and inconsistent identifiers across tables. Additionally, recommendations were made to handle potential KeyError exceptions in the exporter, improve logging for background export failures, and ensure non-negative timeouts during batch writes.

mutianf · 2026-04-29T17:11:24Z

+        write_ind = 0
+        while write_ind < len(series):
+            # find time left for next batch
+            timeout = deadline - time.time() if deadline else gapic_v1.method.DEFAULT


why do we need to calculate the remaining deadline for each batch? I think each batch should just use the default timeout.

The intention here is to be consistent with the OpenTelemetry exporter spec, which passes in a timeout value for exporters to conform to.

I'm not entirely sure what happens if we ignore it, but I can look into it if needed. Does Java not have anything similar?

…csm_3_handlers

daniel-sanche · 2026-07-23T21:32:38Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces client-side metrics collection and exporting to Google Cloud Monitoring for the Bigtable Python client by adding GoogleCloudMetricsHandler and BigtableMetricsExporter. Feedback on the changes highlights several critical issues, including potential KeyError and TypeError exceptions during metric label processing and protobuf serialization, broad exception handling that silently masks export failures, and side-effects from instantiating OpenTelemetry instruments as default arguments. Additionally, corrections are needed in the system tests to resolve mismatches in metric names and type prefixes.

…rics/handlers/opentelemetry.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…rics/handlers/gcp_exporter.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

mutianf · 2026-08-03T18:23:53Z

+        self.meter_provider = MeterProvider(
+            metric_readers=[gcp_reader], views=VIEW_LIST
+        )
+        otel = _OpenTelemetryInstruments(meter_provider=self.meter_provider)


Looks like this should support application has it's own otel instrumentation. Is there an example of how that can be done somewhere?

Yes, I designed the architecture to work well with user-provided otel meter providers, but so far I haven't actually exposed it. I thought we should wait for an ask before making it an official feature

If you want me to add this now, I can. But it might be better as a separate PR

mutianf · 2026-08-03T18:24:48Z

+        super().__init__()
+        self.client = MetricServiceClient(*client_args, **client_kwargs)
+        self.prefix = "bigtable.googleapis.com/internal/client"
+        self.project_id = project_id


I don't think CloudMonitoring client needs to be created with a project id. The project id can be extracted from metrics label ?

Yeah, I think I originally did it this way

because we know the expected project at client init-time

to be more in line with the upstream OTel Exporter

But I don't think there's any reason we need to do it this way. I changed it to extract all project ids from the labels

mutianf · 2026-08-03T18:29:31Z

+            write_ind += max_batch_size
+
+    @staticmethod
+    def _to_point(


are the resouce level attributes (project, instance, table, cluster, zone) removed from the metric data points?

Yes, that was happening here. But I updated this code to be a bit cleaner in this area

mutianf · 2026-08-03T18:33:01Z

+        metric_kind = MetricDescriptor.MetricKind.CUMULATIVE
+        all_series: list[TimeSeries] = []
+        # process each metric from OTel format into Cloud Monitoring format
+        for resource_metric in metrics_data.resource_metrics:


I assume one of these for loops is filtering out bigtable only metrics?

For this handler, we're using an isolated MeterProvider/instruments for bigtable metrics, so we don't actually expect to see any non-bigtable metrics come through.

(we made it so users can also interact with these metrics in their own OTel set-up if we want to expose that, but this pipeline is private)

But also, yes, there is some filtering here, skipping over data that doesn't have the project id set properly

mutianf · 2026-08-03T18:35:05Z

+        # fixed labels sent with each metric update
+        self.shared_labels = {
+            "client_name": f"python-bigtable/{client_version}",
+            "client_uid": client_uid or self._generate_client_uid(),


how is client_uid generated otherwise? maybe we should always generate it? This is really to prevent cloud monitoring to having issue merging the streams. maybe we should move it to the exporter?

how is client_uid generated otherwise? maybe we should always generate it?

Currently, it's always auto-generated, unless explicitly passed in. I don't think we currently need to hard-code it anywhere (outside of possible testing), but if we choose to expose this to end-users, they may want to be able to control this field for their private metric systems

maybe we should move it to the exporter?

This is where we're setting all the other labels, so it feels like a natural place for it. But yeah, if you want to expose these metrics to external OTel set-ups, and don't want them to see this field, we can move it to the GCP-specific subclass instead

daniel-sanche and others added 18 commits April 17, 2026 12:38

moved over instrumentation code

8a3af45

refactored system tests

cf58c57

updated tests

00618f8

fixed lint

ead3bf1

Merge branch 'main' into bigtable_csm_1_basic_instrumentation

af0beba

regenerated files

d0c03d0

added new metrics tests

a78cb52

fixed mtls tests

01f1739

copied over changes

0ab1b05

regenerated sync files

83cbcf0

removed duplicate files

7ca6e11

fixed format

d40fa0a

Merge branch 'bigtable_csm_1_basic_instrumentation' into bigtable_csm…

b94e9d7

…_2_instrumentation_advanced

copied over async changes

394cb09

added missing files

186a69e

removed unneeded grpc_throttling_time

ca0949d

fixed mocking

e2a4190

ran format

f025e3e

gemini-code-assist Bot reviewed Apr 22, 2026

View reviewed changes

mocked out exporter in veneer tests

8668c63

mutianf reviewed Apr 29, 2026

View reviewed changes

daniel-sanche and others added 2 commits May 8, 2026 16:12

Merge branch 'main' into bigtable_csm_2_instrumentation_advanced

742b6a3

Merge branch 'bigtable_csm_2_instrumentation_advanced' into bigtable_…

56c6f95

…csm_3_handlers

Base automatically changed from bigtable_csm_2_instrumentation_advanced to main June 5, 2026 15:54

Merge branch 'main' into bigtable_csm_3_handlers

37892cc

gemini-code-assist Bot reviewed Jul 23, 2026

View reviewed changes

daniel-sanche and others added 3 commits July 23, 2026 17:13

Update packages/google-cloud-bigtable/google/cloud/bigtable/data/_met…

d400e61

…rics/handlers/opentelemetry.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update packages/google-cloud-bigtable/google/cloud/bigtable/data/_met…

3a3ec5c

…rics/handlers/gcp_exporter.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

fixed typo

12d9eab

daniel-sanche added 13 commits July 23, 2026 17:14

removed redundant check

7db7a03

updated buckets

4fe2d19

added logging to metrics exports

194679d

updated deadlines

45ec068

fixed lint

3789937

don't send metrics in emulator mode

7caecfc

mock metrics export for unit tests

da850c5

addressing test issues

0c0f668

fixed tests

de4a42e

remove exporter attribute con client

3900ba5

added assertions for table_id

ca8ffa6

fixed lint

24731d8

fix mypy issues

491c5f9

daniel-sanche changed the title ~~[DRAFT] feat: client side metrics handlers~~ feat: client side metrics handlers Jul 24, 2026

daniel-sanche marked this pull request as ready for review July 28, 2026 22:02

daniel-sanche requested a review from a team as a code owner July 28, 2026 22:02

daniel-sanche and others added 4 commits July 28, 2026 15:02

Merge branch 'main' into bigtable_csm_3_handlers

86a544d

fallback if credentials fail for metrics client

4e46c66

fixed tests

8481cb0

fixed lint

b261466

mutianf reviewed Aug 3, 2026

View reviewed changes

daniel-sanche added 6 commits August 3, 2026 17:21

Merge branch 'main' into bigtable_csm_3_handlers

0950159

use passed in project for all exporter tasks

92156ac

cleaned up attribute parsing

f91df37

fallback to no handler, instead of Otel

c0fad79

fixed lint/mypy

0b06c86

regenerated sync client

61fac3a

daniel-sanche changed the title ~~feat: client side metrics handlers~~ feat(bigtable): client side metrics handlers Aug 7, 2026

daniel-sanche added 2 commits August 7, 2026 09:51

addressing system test issues

4982602

fixed handler setup in metrics tests

236f493

Uh oh!

Conversation

daniel-sanche commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

daniel-sanche commented Jul 23, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

daniel-sanche commented Apr 22, 2026 •

edited

Loading