Tracking issue here: https://github.com/coder/internal/issues/1009
To summarize, the current version of this query selects from
`workspace_agent_stats` twice. The expensive portion of this query is
the bitmap heap scan we have to do for each of these selects. We can
easily cut the cost of this query by 40-50% by cutting this down to a
single select, and using those rows for both sets of calculations.
Eliminating the heap scan itself would require a follow up PR to
introduce a new index. Blink helped with the rewrite of the query.
The current plan looks like this:
```
Nested Loop (cost=6101.64..6101.69 rows=1 width=64) (actual time=11.782..11.787 rows=1 loops=1)
-> Aggregate (cost=2996.17..2996.19 rows=1 width=32) (actual time=3.356..3.357 rows=1 loops=1)
-> Bitmap Heap Scan on workspace_agent_stats (cost=54.80..2992.86 rows=440 width=24) (actu
al time=0.346..2.927 rows=818 loops=1)
Recheck Cond: (created_at > (now() - '00:15:00'::interval))
Filter: (connection_median_latency_ms > '0'::double precision)
Rows Removed by Filter: 1070
Heap Blocks: exact=486
-> Bitmap Index Scan on idx_agent_stats_created_at (cost=0.00..54.69 rows=1368 width
=0) (actual time=0.241..0.241 rows=1888 loops=1)
Index Cond: (created_at > (now() - '00:15:00'::interval))
-> Aggregate (cost=3105.47..3105.49 rows=1 width=32) (actual time=8.418..8.420 rows=1 loops=1)
-> Subquery Scan on a (cost=3060.95..3105.39 rows=7 width=32) (actual time=7.851..8.394 ro
ws=63 loops=1)
Filter: (a.rn = 1)
-> WindowAgg (cost=3060.95..3088.29 rows=1368 width=209) (actual time=7.850..8.382 r
ows=63 loops=1)
Run Condition: (row_number() OVER (?) <= 1)
-> Sort (cost=3060.93..3064.35 rows=1368 width=56) (actual time=7.836..8.036 r
ows=1888 loops=1)
Sort Key: workspace_agent_stats_1.agent_id, workspace_agent_stats_1.create
d_at DESC
Sort Method: quicksort Memory: 181kB
-> Bitmap Heap Scan on workspace_agent_stats workspace_agent_stats_1 (co
st=55.03..2989.67 rows=1368 width=56) (actual time=0.388..2.096 rows=1888 loops=1)
Recheck Cond: (created_at > (now() - '00:15:00'::interval))
Heap Blocks: exact=486
-> Bitmap Index Scan on idx_agent_stats_created_at (cost=0.00..54.
69 rows=1368 width=0) (actual time=0.295..0.295 rows=1888 loops=1)
Index Cond: (created_at > (now() - '00:15:00'::interval))
Planning Time: 2.350 ms
Execution Time: 13.152 ms
(24 rows)
```
The new plan looks like this
```
Aggregate (cost=2966.96..2966.98 rows=1 width=64) (actual time=3.812..3.814 rows=1 loops=1)
-> WindowAgg (cost=2891.96..2916.94 rows=1250 width=88) (actual time=2.696..3.412 rows=1890 loop
s=1)
-> Sort (cost=2891.94..2895.06 rows=1250 width=80) (actual time=2.686..2.780 rows=1890 loo
ps=1)
Sort Key: workspace_agent_stats.agent_id, workspace_agent_stats.created_at DESC
Sort Method: quicksort Memory: 226kB
-> Bitmap Heap Scan on workspace_agent_stats (cost=50.11..2827.64 rows=1250 width=80
) (actual time=0.218..1.551 rows=1890 loops=1)
Recheck Cond: (created_at > (now() - '00:15:00'::interval))
Heap Blocks: exact=474
-> Bitmap Index Scan on idx_agent_stats_created_at (cost=0.00..49.80 rows=1250
width=0) (actual time=0.146..0.147 rows=1890 loops=1)
Index Cond: (created_at > (now() - '00:15:00'::interval))
Planning Time: 0.534 ms
Execution Time: 3.969 ms
(12 rows)
```
If we compare the results of the query they're similar enough that any
differences can be attributed to slightly different timestamps for
`now()` in the version of the query I am using to generate results for
comparison:
```
workspace_rx_bytes | workspace_tx_bytes | workspace_connection_latency_50 | workspace_connection_latency_95 | session_count_vscode | session_count_ssh | session_count_jetbrains | session_count_reconnecting_pty
--------------------+--------------------+---------------------------------+---------------------------------+----------------------+-------------------+-------------------------+--------------------------------
15263563 | 74555854 | 47.933 | 250.5522 | 239 | 59 | 3 | 3
(1 row)
workspace_rx_bytes | workspace_tx_bytes | workspace_connection_latency_50 | workspace_connection_latency_95 | session_count_vscode | session_count_ssh | session_count_jetbrains | session_count_reconnecting_pty
--------------------+--------------------+---------------------------------+---------------------------------+----------------------+-------------------+-------------------------+--------------------------------
15295819 | 74598410 | 47.933 | 250.5522 | 239 | 59 | 3 | 3
```
---------
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Fixes#13430.
The test for purging old workspace agent stats from the DB was consistently failing when ran with Postgres towards the end of the month, but not with the in-memory DB.
This was because month intervals are calculated differently for `time.Time` and the `interval` type in Postgres:
```
ethan=# SELECT
'2024-08-30'::DATE AS original_date,
('2024-08-30'::DATE - INTERVAL '6 months') AS sub_date;
original_date | sub_date
---------------+---------------------
2024-08-30 | 2024-02-29 00:00:00
(1 row)
```
Using `func (t Time) AddDate(years int, months int, days int) Time`, where `months` is `-6`:
```
Original: 2024-08-30 00:00:00 +0000 UTC
6 Months Earlier: 2024-03-01 00:00:00 +0000 UTC
```
Since 6 months was chosen arbitrarily, we should be able to change it to 180 days, to remove any ambiguity between the in-memory DB, and the Postgres DB. The alternative solution would involve implementing Postgres' month interval algorithm in Go.
The UI only shows stats as old as 168 days (24 weeks), so a frontend change isn't required for the extra days of stats we lose in some cases.
* chore: remove InsertWorkspaceAgentStat query
InsertWorkspaceAgentStats (batch) exists. We only used the singular in
a single unit test place. Removing the single for the batch, reducing
the interface size.
* fix: don't make session counts cumulative
This made for some weird tracking... we want the point-in-time
number of counts!
* Add databasefake query for getting agent stats
* Add deployment stats endpoint
* The query... works?!?
* Fix aggregation query
* Select from multiple tables instead
* Fix continuous stats
* Increase period of stat refreshes
* Add workspace counts to deployment stats
* fmt
* Add a slight bit of responsiveness
* Fix template version editor overflow
* Add refresh button
* Fix font family on button
* Fix latest stat being reported
* Revert agent conn stats
* Fix linting error
* Fix tests
* Fix gen
* Fix migrations
* Block on sending stat updates
* Add test fixtures
* Fix response structure
* make gen
* chore: convert workspace agent stats from json to table
* chore: convert agent stats to use a table
Backwards compatibility becomes hard when all agent stats are in a JSON blob.
We also want to query this table for new agents that are failing health checks
so we can display it in the UI.
* Fix migration using default values
Callum Styan