Setup and document continuous deployment / orchestration #3
Labels
No Label
Kind/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
No Milestone
No project
1 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: TEDective/etl#3
Loading…
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The goal is to automatically run the ETL pipeline on each push to a yet to be created
prod
branch. Once a commit is pushed to theprod
branch, all the necessary steps should be executed on our main workload serverholmes
.To achieve this, AFAICU we can either:
or
prod
branch. This is probably harder to maintain in the long run.Setup continuous deploymentto Setup and document continuous deploymentWhen switching over to prefect and using
holmes
as our worker, we'd have to pay particular attention to how concurrency over here will be handled. Probably with something like this.This might be good inspiration and here is an intro video for prefect.
Setup and document continuous deploymentto Setup and document continuous deployment / orchestrationI ditched prefect and started using luigi instead. Much simpler and better geared for our use case IMHO.