Sampreeth Avvari
Resume
Read the story

Architecture

Flags bad CT scans before design.$62 in October, on CPU.

The lab's case system exports each CBCT case zip to an SFTP box, a new zip in the bucket fires Eventarc, and a frozen dental encoder on Cloud Run CPU scores it and sends the verdict PDF back through the same box. Labeled Drive folders become training tensors each morning, and the testing team checks the model on real studies in a review app with its own scoreboard.

2.2 s

per pass of the frozen encoder on CPU with OpenVINO, measured before deploy

20 scans

in a frozen holdout the trainer refuses to train on

38 days

from the first commit on Mar 25 to Model H, the model it runs today, on May 1

26 of 30

bad studies the testing team found that the model also flagged, Aug 19 to Oct 28

New York --:-- simulated

A new scan
Hybridge CBCT Scan Validator The lab's case system exports each CBCT case zip to an SFTP box, a new zip in the bucket fires Eventarc, and a frozen dental encoder on Cloud Run CPU scores it and sends the verdict PDF back through the same box. Labeled Drive folders become training tensors each morning, and the testing team checks the model on real studies in a review app with its own scoreboard. Google Drive labeled scan folders training-etl-sync Cloud Run job, 6 AM Head training RTX 4080, frozen encoder Google IAP Hybridge accounts only MagicTouch DLCPM the lab case system cbct-sftp VM chrooted SFTP box Scan bucket Cloud Storage cbct-review Cloud Run, test app Testing team real studies cbct-validator Cloud Run, OpenVINO CPU Eventarc fires on each new zip BigQuery verdicts, append only zips tensors model head SFTP case zip new zip POST /event verdict PDF upload, judge sign in signed upload rows GCP cbct-scan-validation, us-east1

Hover a part to trace what it touches. Click it for its notes.

  • Built by me
  • Managed by Google
  • Outside system
  • Main data path
  • On demand
  • Credentials cross

When it breaks

What each failure does, and what it does not do.

  1. Eventarc fires twice

    The first run moves the zip to processed/, so the second finds nothing to score. Skipped events answer 200 and real failures 500, so only real failures retry.

    scan_val/inference/service.py:326

  2. One upload of two fails

    The SFTP box deletes its copy only after the bucket and Drive both have the file. Until then the file waits in drop for the next retry.

    scripts/sftp_to_gcs_and_drive.py:8

  3. A scan will not convert

    The ETL tries twice, then moves the zip to a Manual review folder, logs the error and emails me.

    scripts/training_data_sync.py:16

What I tuned

Small choices that keep it fast and cheap.

  1. Daily at 6 AM

    The training ETL woke every 5 minutes on 4 CPUs and 16 GiB, 53,086 runs since April. On October 28 it moved to once a day at 6 AM.

    Cloud Scheduler job training-etl-sync-trigger

  2. Encoder baked, head swapped

    The encoder is compiled into the image per release. The small head loads from the bucket at cold start, so a retrain ships without a rebuild.

    docs/PHASE1_PRODUCTION_PLAN.md:31

  3. OpenVINO on CPU

    2.20 seconds per encoder pass with OpenVINO, 2.56 with PyTorch and 3.93 with ONNX Runtime alone. The validator runs on CPU and scales to zero.

    docs/PATH_A_LATENCY_SPIKE_FINDINGS.md:36

Guards

Rules the code enforces, not rules people remember.

  1. The holdout never trains

    Training stops before it starts if any of the 20 holdout studies appears in its labels. The split puts all 20 in the test set.

    scripts/run_train_v2.py:739

  2. SFTP only, chrooted

    The lab's user is locked into its own folder and can only speak SFTP. It gets no shell and no port forwarding.

    scripts/sftp-cloud-init.yaml:34

  3. Write back is a switch

    Pushing a verdict into the lab system needs an explicit flag on the service. With the flag off, the validator still scores and reports.

    scan_val/inference/dlcpm_writeback.py:41

What it costs

Measured from the Google Cloud billing export, October 1 to 28, net of credits. About three quarters is the training ETL, which woke every 5 minutes until October 28 and now runs once a day. The drop is not measured yet.

$62in October

  • $45.54 Cloud Run jobs, the training ETL
  • $11.23 SFTP VM, always on
  • $4.07 Image registry and storage
  • $0.92 Services, secrets, scheduler