Skip to content

Update gcs-archive.md #137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
qjhart wants to merge 1 commit into
base: sandbox
Choose a base branch
from
Open

Update gcs-archive.md #137

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 30 additions & 6 deletions docs/data-development/gcs-archive.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,30 +15,54 @@ docker context use sandbox.dams

## Authenticate

Make sure the service account is added to data dev container
Make sure the service account key is added to data dev container. For example, if your key is in a google secret, then

```bash
docker cp [path-to-service-account]/service-account.json [collection-name]-import:/
# Get your secret / copy to your machine
key=dams-collection-archiver-key
col=[collection-name]
gcloud secrets versions access latest --secret=${key} > key.json

docker exec -it [collection-name]-import bash
docker cp key.json ${col}-import:/etc/service-account.json

gcloud auth activate-service-account --key-file=/service-account.json
docker exec -it ${col}-import bash
```

Then inside the import container...

```bash
gcloud auth activate-service-account --key-file=/etc/service-account.json
```


## Sync data

### To bucket from container
### To bucket From container

```bash
cd /data
gsutil -m rsync -c -J -r -y "^(\.git|.*\.jsonld\.json)$" -n /data gs://dams-collections/[collection-name]
```

or the above combined more succintly, run all this on your local machine

```bash
col=wine-labels # for example
key=dams-collection-archiver-key
docker context use sandbox.dams
gcloud secrets versions access latest --secret=${key} > key.json
docker cp key.json ${col}-import:/etc/service-account.json
docker exec -i -t ${col}-import bash -c "{ gcloud auth activate-service-account --key-file=/etc/service-account.json; gcloud storage rsync --continue-on-error --gzip-in-flight-all --recursive --exclude='^(\.git|.*\.jsonld\.json)\$' --dry-run /data gs://dams-collections/${col}; rm /etc/service-account.json}"

```



### From bucket to container

```bash
gsutil -m rsync -c -J -r -y ".*\.jsonld\.json$" gs://dams-collections/[collection-name] /data
```

`-n` flag is dry run. Remove to actually sync. Note this doesn't delete. See docs here: https://cloud.google.com/storage/docs/gsutil/commands/rsync
`-n` flag is dry run. Remove to actually sync. Note this doesn't delete. See docs here: https://cloud.google.com/storage/docs/gsutil/commands/rsync