------------------------------------ Dashboard Generation Developer Guide ------------------------------------ *Overview* The empty-room data quality dashboards are generated from the BIDS empty-room dataset hosted on NYU Box. The scripts live under ``docs/source/scripts/dashboard-generating-scripts`` and are run by the ``Empty-room Data Quality Dashboard`` GitHub Action (``.github/workflows/empty-room-dashboard.yml``) every day at 05:00 UTC. The Action downloads any recording not yet present in the metrics CSV, computes the metrics, regenerates the figures and commits the resulting CSV, HTML and PNG files back to ``main`` (the commit carries ``[skip ci]``; the daily docs deploy at 06:00 UTC publishes the refreshed files to the website). The docs build itself never talks to Box, so a Box or network problem can never break the website build. The stack being used comprises: - data source: NYU Box (BIDS empty-room dataset) - compute: MNE, NumPy, SciPy, pandas - figures: Plotly (interactive HTML) and Matplotlib (sensor-topography PNG) - automation: GitHub Actions - rendering: Sphinx, deployed to Cloudflare Pages (public and internal sites) The scripts are: - ``config.py`` - all paths, Box folder identifiers, the per system definitions (acq entity, file extension, thresholds, output CSV paths) and the analysis parameters (window, PSD settings, line frequency). - paths are anchored to the file location, so the scripts run the same from the repository root, from ``docs/source`` or from the Action. - ``utilities.py`` - authenticates to Box from a single JWT settings JSON (``BOX_CLIENT_SDK_CONFIG``); - resolves the empty-room dataset folder (see Authentication below), walks it recursively (``sub-emptyroom/ses-*/meg``) and classifies recordings by the BIDS ``acq`` entity (``acq-kit`` gives ``.con``, ``acq-opm`` gives ``.fif``); derivatives carrying a ``proc-`` entity are skipped so raw room noise is tracked consistently; - computes per sensor RMS, PSD noise floor and line-noise, plus the summary average, variance and maximum, and writes the per recording and per sensor CSVs; - renders the sensor-topography maps (the per recording PNGs, the recent-sessions grid and an interactive hover version with a session dropdown, built from the per channel CSV plus the stored sensor positions), the sensor by session heatmap, the amplitude spectrum and the metric time-series figures. Sensor positions are written to ``-sensor-positions.csv`` whenever a recording is processed, so the interactive map can also be rebuilt by runs that download nothing new; for KIT the builtin ``KIT-AD`` layout serves as fallback until that file exists. - ``update_data_quality_dashboards.py`` - the entry point. Run it with no arguments for a Box run, or with ``--local-dir`` to process a local BIDS tree for testing (no Box credentials needed). ``--systems kit,opm`` selects systems. - ``generate_system_status_dashboards.py`` - builds the system status cards (KIT, OPM, QD helium, Vpixx) from the status CSVs under ``docs/source/data/system-status-dashboards``. The output is small self-contained HTML with no Plotly dependency; edit the CSVs to record a new week. *Authentication and configuration* In CI the credentials come from repository Actions secrets: - ``BOX_CLIENT_SDK_CONFIG``: the complete Box JWT app settings JSON, including the ``appAuth`` private key block (a JSON downloaded at keypair generation time; a later re-download from the Box console has an empty ``appAuth`` and will not work). - ``BOX_EMPTYROOM_FOLDER_ID`` (recommended): the folder ID of the dataset root that contains ``sub-emptyroom``, taken from the folder's URL in the Box web app. - ``BOX_MEG_DATA_PARENT_FOLDER_ID``: fallback starting point. When the direct ID is not set, the resolver walks ``Data/empty-room`` from this parent, then tries ``empty-room`` directly, then ``MEG/Data/empty-room``, and finally accepts the parent itself if it already contains the BIDS subject folders. - ``DASHBOARD_PUSH_TOKEN``: an administrator personal access token used by the workflow to push the refreshed artifacts to the protected ``main`` branch. For a manual run, place the same JWT JSON in ``box_config.json`` next to the scripts (this file is git-ignored and must never be committed). .. code-block:: python import json, os from boxsdk import Client, JWTAuth settings = json.loads(os.environ["BOX_CLIENT_SDK_CONFIG"]) client = Client(JWTAuth.from_settings_dictionary(settings)) *Running locally* .. code-block:: bash pip install "boxsdk[jwt]" mne plotly pandas numpy scipy matplotlib cd docs/source/scripts/dashboard-generating-scripts # Offline test against a local BIDS tree (no Box needed): python update_data_quality_dashboards.py --local-dir /path/to/empty-room --systems kit # Full Box run (requires BOX_CLIENT_SDK_CONFIG): python update_data_quality_dashboards.py --systems kit,opm *Outputs* - ``docs/source/data/data-quality-dashboards/-empty-room-metrics.csv`` : one row per recording. - ``docs/source/data/data-quality-dashboards/-per-channel-metrics.csv`` : one row per sensor per recording. - ``docs/source/data/data-quality-dashboards/-empty-room-metrics-recent.csv`` : the always-visible most recent rows of the data quality page's table. - ``docs/source/data/data-quality-dashboards/-empty-room-metrics-display.csv`` : the full history table shown in the collapsed dropdown. - ``docs/source/data/data-quality-dashboards/-sensor-positions.csv`` : 2D sensor layout used by the interactive topomap. - ``docs/source/_static/2-data-quality-dashboards/_*.html`` : interactive figures. - ``docs/source/_static/2-data-quality-dashboards/_topomap_recent.png`` : sensor-topography maps. *Adding or changing a metric* Add the computation in ``compute_metrics`` in ``utilities.py``, add the column to ``METRIC_COLUMNS`` (and to ``DISPLAY_COLUMNS`` if it should appear in the table), document it in ``docs/source/data/data-quality-dashboards/noise_metrics.csv``, and add a figure builder if a plot is wanted.