Skip to content

Commit 997239a

Browse files
rjsparksjennifer-richards
rjsparks
and
jennifer-richards
authored
feat: write objects to blob storage (#8557)
* feat: basic blobstore infrastructure for dev * refactor: (broken) attempt to put minio console behind nginx * feat: initialize blobstore with boto3 * fix: abandon attempt to proxy minio. Use docker compose instead. * feat: beginning of blob writes * feat: storage utilities * feat: test buckets * chore: black * chore: remove unused import * chore: avoid f string when not needed * fix: inform all settings files about blobstores * fix: declare types for some settings * ci: point to new target base * ci: adjust test workflow * fix: give the tests debug environment a blobstore * fix: "better" name declarations * ci: use devblobstore container * chore: identify places to write to blobstorage * chore: remove unreachable code * feat: store materials * feat: store statements * feat: store status changes * feat: store liaison attachments * feat: store agendas provided with Interim session requests * chore: capture TODOs * feat: store polls and chatlogs * chore: remove unneeded TODO * feat: store drafts on submit and post * fix: handle storage during doc expiration and resurrection * fix: mirror an unlink * chore: add/refine TODOs * feat: store slide submissions * fix: structure slide test correctly * fix: correct sense of existence check * feat: store some indexes * feat: BlobShadowFileSystemStorage * feat: shadow floorplans / host logos to the blob * chore: remove unused import * feat: strip path from blob shadow names * feat: shadow photos / thumbs * refactor: combine photo and photothumb blob kinds The photos / thumbs were already dropped in the same directory, so let's not add a distinction at this point. * style: whitespace * refactor: use kwargs consistently * chore: migrations * refactor: better deconstruct(); rebuild migrations * fix: use new class in mack patch * chore: add TODO * feat: store group index documents * chore: identify more TODO * feat: store reviews * fix: repair merge * chore: remove unnecessary TODO * feat: StoredObject metadata * fix: deburr some debugging code * fix: only set the deleted timestamp once * chore: correct typo * fix: get_or_create vs get and test * fix: avoid the questionable is_seekable helper * chore: capture future design consideration * chore: blob store cfg for k8s * chore: black * chore: copyright * ci: bucket name prefix option + run Black Adds/uses DATATRACKER_BLOB_STORE_BUCKET_PREFIX option. Other changes are just Black styling. * ci: fix typo in bucket name expression * chore: parameters in app-configure-blobstore Allows use with other blob stores. * ci: remove verify=False option * fix: don't return value from __init__ * feat: option to log timing of S3Storage calls * chore: units * fix: deleted->null when storing a file * style: Black * feat: log as JSON; refactor to share code; handle exceptions * ci: add ietf_log_blob_timing option for k8s * test: --no-manage-blobstore option for running tests * test: use blob store settings from env, if set * test: actually set a couple more storage opts * feat: offswitch (#8541) * feat: offswitch * fix: apply ENABLE_BLOBSTORAGE to BlobShadowFileSystemStorage behavior * chore: log timing of blob reads * chore: import Config from botocore.config * chore(deps): import boto3-stubs / botocore botocore is implicitly imported, but make it explicit since we refer to it directly * chore: drop type annotation that mypy loudly ignores * refactor: add storage methods via mixin Shares code between Document and DocHistory without putting it in the base DocumentInfo class, which lacks the name field. Also makes mypy happy. * feat: add timeout / retry limit to boto client * ci: let k8s config the timeouts via env * chore: repair merge resolution typo * chore: tweak settings imports * chore: simplify k8s/settings_local.py imports --------- Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
1 parent e71272f commit 997239a

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+1484
-116
lines changed

‎.devcontainer/docker-compose.extend.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,10 @@ services:
1414
# - datatracker-vscode-ext:/root/.vscode-server/extensions
1515
# Runs app on the same network as the database container, allows "forwardPorts" in devcontainer.json function.
1616
network_mode: service:db
17+
blobstore:
18+
ports:
19+
- '9000'
20+
- '9001'
1721

1822
volumes:
1923
datatracker-vscode-ext:

‎.github/workflows/tests.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@ jobs:
2828
services:
2929
db:
3030
image: ghcr.io/ietf-tools/datatracker-db:latest
31+
blobstore:
32+
image: ghcr.io/ietf-tools/datatracker-devblobstore:latest
3133

3234
steps:
3335
- uses: actions/checkout@v4

‎README.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,23 @@ Nightly database dumps of the datatracker are available as Docker images: `ghcr.
106106

107107
> Note that to update the database in your dev environment to the latest version, you should run the `docker/cleandb` script.
108108

109+
### Blob storage for dev/test
110+
111+
The dev and test environments use [minio](https://github.com/minio/minio) to provide local blob storage. See the settings files for how the app container communicates with the blobstore container. If you need to work with minio directly from outside the containers (to interact with its api or console), use `docker compose` from the top level directory of your clone to expose it at an ephemeral port.
112+
113+
```
114+
$ docker compose port blobstore 9001
115+
0.0.0.0:<some ephemeral port>
116+
117+
$ curl -I http://localhost:<some ephemeral port>
118+
HTTP/1.1 200 OK
119+
...
120+
```
121+
122+
123+
The minio container exposes the minio api at port 9000 and the minio console at port 9001
124+
125+
109126
### Frontend Development
110127
111128
#### Intro

‎dev/deploy-to-container/settings_local.py‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
# Copyright The IETF Trust 2007-2019, All Rights Reserved
22
# -*- coding: utf-8 -*-
33

4-
from ietf.settings import * # pyflakes:ignore
4+
from ietf.settings import * # pyflakes:ignore
5+
from ietf.settings import STORAGES, MORE_STORAGE_NAMES, BLOBSTORAGE_CONNECT_TIMEOUT, BLOBSTORAGE_READ_TIMEOUT, BLOBSTORAGE_MAX_ATTEMPTS
6+
import botocore.config
57

68
ALLOWED_HOSTS = ['*']
79

@@ -79,3 +81,22 @@
7981

8082
# OIDC configuration
8183
SITE_URL = 'https://__HOSTNAME__'
84+
85+
for storagename in MORE_STORAGE_NAMES:
86+
STORAGES[storagename] = {
87+
"BACKEND": "ietf.doc.storage_backends.CustomS3Storage",
88+
"OPTIONS": dict(
89+
endpoint_url="http://blobstore:9000",
90+
access_key="minio_root",
91+
secret_key="minio_pass",
92+
security_token=None,
93+
client_config=botocore.config.Config(
94+
signature_version="s3v4",
95+
connect_timeout=BLOBSTORAGE_CONNECT_TIMEOUT,
96+
read_timeout=BLOBSTORAGE_READ_TIMEOUT,
97+
retries={"total_max_attempts": BLOBSTORAGE_MAX_ATTEMPTS},
98+
),
99+
verify=False,
100+
bucket_name=f"test-{storagename}",
101+
),
102+
}

‎dev/diff/settings_local.py‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
# Copyright The IETF Trust 2007-2019, All Rights Reserved
22
# -*- coding: utf-8 -*-
33

4-
from ietf.settings import * # pyflakes:ignore
4+
from ietf.settings import * # pyflakes:ignore
5+
from ietf.settings import STORAGES, MORE_STORAGE_NAMES, BLOBSTORAGE_CONNECT_TIMEOUT, BLOBSTORAGE_READ_TIMEOUT, BLOBSTORAGE_MAX_ATTEMPTS
6+
import botocore.config
57

68
ALLOWED_HOSTS = ['*']
79

@@ -66,3 +68,22 @@
6668
SLIDE_STAGING_PATH = 'test/staging/'
6769

6870
DE_GFM_BINARY = '/usr/local/bin/de-gfm'
71+
72+
for storagename in MORE_STORAGE_NAMES:
73+
STORAGES[storagename] = {
74+
"BACKEND": "ietf.doc.storage_backends.CustomS3Storage",
75+
"OPTIONS": dict(
76+
endpoint_url="http://blobstore:9000",
77+
access_key="minio_root",
78+
secret_key="minio_pass",
79+
security_token=None,
80+
client_config=botocore.config.Config(
81+
signature_version="s3v4",
82+
connect_timeout=BLOBSTORAGE_CONNECT_TIMEOUT,
83+
read_timeout=BLOBSTORAGE_READ_TIMEOUT,
84+
retries={"total_max_attempts": BLOBSTORAGE_MAX_ATTEMPTS},
85+
),
86+
verify=False,
87+
bucket_name=f"test-{storagename}",
88+
),
89+
}

‎dev/tests/docker-compose.debug.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,5 +28,8 @@ services:
2828
volumes:
2929
- postgresdb-data:/var/lib/postgresql/data
3030

31+
blobstore:
32+
image: ghcr.io/ietf-tools/datatracker-devblobstore:latest
33+
3134
volumes:
3235
postgresdb-data:

‎dev/tests/settings_local.py‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
# Copyright The IETF Trust 2007-2019, All Rights Reserved
22
# -*- coding: utf-8 -*-
33

4-
from ietf.settings import * # pyflakes:ignore
4+
from ietf.settings import * # pyflakes:ignore
5+
from ietf.settings import STORAGES, MORE_STORAGE_NAMES, BLOBSTORAGE_CONNECT_TIMEOUT, BLOBSTORAGE_READ_TIMEOUT, BLOBSTORAGE_MAX_ATTEMPTS
6+
import botocore.config
57

68
ALLOWED_HOSTS = ['*']
79

@@ -65,3 +67,22 @@
6567
SLIDE_STAGING_PATH = 'test/staging/'
6668

6769
DE_GFM_BINARY = '/usr/local/bin/de-gfm'
70+
71+
for storagename in MORE_STORAGE_NAMES:
72+
STORAGES[storagename] = {
73+
"BACKEND": "ietf.doc.storage_backends.CustomS3Storage",
74+
"OPTIONS": dict(
75+
endpoint_url="http://blobstore:9000",
76+
access_key="minio_root",
77+
secret_key="minio_pass",
78+
security_token=None,
79+
client_config=botocore.config.Config(
80+
signature_version="s3v4",
81+
connect_timeout=BLOBSTORAGE_CONNECT_TIMEOUT,
82+
read_timeout=BLOBSTORAGE_READ_TIMEOUT,
83+
retries={"total_max_attempts": BLOBSTORAGE_MAX_ATTEMPTS},
84+
),
85+
verify=False,
86+
bucket_name=f"test-{storagename}",
87+
),
88+
}

‎docker-compose.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ services:
1515
depends_on:
1616
- db
1717
- mq
18+
- blobstore
1819

1920
ipc: host
2021

@@ -83,6 +84,14 @@ services:
8384
- .:/workspace
8485
- app-assets:/assets
8586

87+
blobstore:
88+
image: ghcr.io/ietf-tools/datatracker-devblobstore:latest
89+
restart: unless-stopped
90+
volumes:
91+
- "minio-data:/data"
92+
93+
94+
8695
# Celery Beat is a periodic task runner. It is not normally needed for development,
8796
# but can be enabled by uncommenting the following.
8897
#
@@ -106,3 +115,4 @@ services:
106115
volumes:
107116
postgresdb-data:
108117
app-assets:
118+
minio-data:

‎docker/app.Dockerfile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,8 @@ RUN rm -rf /tmp/library-scripts
4343
# Copy the startup file
4444
COPY docker/scripts/app-init.sh /docker-init.sh
4545
COPY docker/scripts/app-start.sh /docker-start.sh
46-
RUN sed -i 's/\r$//' /docker-init.sh && chmod +x /docker-init.sh
47-
RUN sed -i 's/\r$//' /docker-start.sh && chmod +x /docker-start.sh
46+
RUN sed -i 's/\r$//' /docker-init.sh && chmod +rx /docker-init.sh
47+
RUN sed -i 's/\r$//' /docker-start.sh && chmod +rx /docker-start.sh
4848

4949
# Fix user UID / GID to match host
5050
RUN groupmod --gid $USER_GID $USERNAME \

‎docker/configs/settings_local.py‎

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
1-
# Copyright The IETF Trust 2007-2019, All Rights Reserved
1+
# Copyright The IETF Trust 2007-2025, All Rights Reserved
22
# -*- coding: utf-8 -*-
33

4-
from ietf.settings import * # pyflakes:ignore
4+
from ietf.settings import * # pyflakes:ignore
5+
from ietf.settings import STORAGES, MORE_STORAGE_NAMES, BLOBSTORAGE_CONNECT_TIMEOUT, BLOBSTORAGE_READ_TIMEOUT, BLOBSTORAGE_MAX_ATTEMPTS
6+
import botocore.config
57

68
ALLOWED_HOSTS = ['*']
79

8-
from ietf.settings_postgresqldb import DATABASES # pyflakes:ignore
10+
from ietf.settings_postgresqldb import DATABASES # pyflakes:ignore
911

1012
IDSUBMIT_IDNITS_BINARY = "/usr/local/bin/idnits"
1113
IDSUBMIT_STAGING_PATH = "/assets/www6s/staging/"
@@ -37,6 +39,25 @@
3739
# DEV_TEMPLATE_CONTEXT_PROCESSORS = [
3840
# 'ietf.context_processors.sql_debug',
3941
# ]
42+
for storagename in MORE_STORAGE_NAMES:
43+
STORAGES[storagename] = {
44+
"BACKEND": "ietf.doc.storage_backends.CustomS3Storage",
45+
"OPTIONS": dict(
46+
endpoint_url="http://blobstore:9000",
47+
access_key="minio_root",
48+
secret_key="minio_pass",
49+
security_token=None,
50+
client_config=botocore.config.Config(
51+
signature_version="s3v4",
52+
connect_timeout=BLOBSTORAGE_CONNECT_TIMEOUT,
53+
read_timeout=BLOBSTORAGE_READ_TIMEOUT,
54+
retries={"total_max_attempts": BLOBSTORAGE_MAX_ATTEMPTS},
55+
),
56+
verify=False,
57+
bucket_name=storagename,
58+
),
59+
}
60+
4061

4162
DOCUMENT_PATH_PATTERN = '/assets/ietfdata/doc/{doc.type_id}/'
4263
INTERNET_DRAFT_PATH = '/assets/ietf-ftp/internet-drafts/'

0 commit comments

Comments
 (0)