TEDective/etl
11
3
Fork 0
Code Issues 20 Pull Requests Actions Packages Projects Releases Wiki Activity

Setup and document continuous deployment / orchestration #3

New Issue
Open
opened 2024-02-05 07:46:15 +00:00 by linus · 3 comments
linus commented 2024-02-05 07:46:15 +00:00
Owner
Copy Link

The goal is to automatically run the ETL pipeline on each push to a yet to be created prod branch. Once a commit is pushed to the prod branch, all the necessary steps should be executed on our main workload server holmes.

To achieve this, AFAICU we can either:

  • A full-blown solution like prefect (would probably require a fair bit of restructuring the code but maybe that's a good thing)

or

  • The deployer user on Gitea actions to also have rights to run the ingestion/deduplication scripts and then set that up via Gitea Actions to run on every push to the prod branch. This is probably harder to maintain in the long run.
The goal is to automatically run the ETL pipeline on each push to a yet to be created `prod` branch. Once a commit is pushed to the `prod` branch, all the necessary steps should be executed on our main workload server `holmes`. To achieve this, AFAICU we can either: - A full-blown solution like [prefect](https://www.prefect.io/) (would probably require a fair bit of restructuring the code but maybe that's a good thing) or - The deployer user on Gitea actions to also have rights to run the ingestion/deduplication scripts and then set that up via Gitea Actions to run on every push to the `prod` branch. This is probably harder to maintain in the long run.
linus added the
Kind/Feature
Kind/Testing
 labels 2024-02-05 17:25:58 +00:00
linus self-assigned this 2024-02-06 10:04:43 +00:00
linus changed title from Setup continuous deployment to Setup and document continuous deployment 2024-02-06 10:17:27 +00:00
linus added
Kind/Documentation
 and removed
Kind/Testing
 labels 2024-02-06 10:17:35 +00:00
linus commented 2024-02-06 14:52:25 +00:00
Author
Owner
Copy Link

When switching over to prefect and using holmes as our worker, we'd have to pay particular attention to how concurrency over here will be handled. Probably with something like this.

When switching over to prefect and using `holmes` as our [worker](https://docs.prefect.io/latest/concepts/work-pools/), we'd have to pay particular attention to how concurrency [over here](https://git.fsfe.org/TEDective/etl/src/commit/7629fbb117d506fc691d21674ba0c291284fb71c/tedective_parser/parser.py#L277-L280) will be handled. Probably with something like [this](https://docs.prefect.io/latest/api-ref/prefect/task-runners/#prefect.task_runners.ConcurrentTaskRunner).
linus commented 2024-02-06 14:55:48 +00:00
Author
Owner
Copy Link

This might be good inspiration and here is an intro video for prefect.

[This](https://github.com/investigativedata/investigraph-etl) might be good inspiration and [here](https://www.youtube.com/watch?v=D5DhwVNHWeU) is an intro video for prefect.
micgor32 was assigned by linus 2024-02-06 14:55:55 +00:00
linus changed title from Setup and document continuous deployment to Setup and document continuous deployment / orchestration 2024-02-06 15:13:52 +00:00
linus commented 2024-02-13 13:42:32 +00:00
Author
Owner
Copy Link

I ditched prefect and started using luigi instead. Much simpler and better geared for our use case IMHO.

I ditched prefect and started using luigi instead. Much simpler and better geared for our use case IMHO.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Kind/Breaking
Breaking change that won't be backward compatible
Kind/Bug
Something is not working
Kind/Documentation
Documentation changes
Kind/Enhancement
Improve existing functionality
Kind/Feature
New functionality
Kind/Security
This is security issue
Kind/Testing
Issue or pull request related to testing
Priority
Critical
The priority is critical
Priority
High
The priority is high
Priority
Low
The priority is low
Priority
Medium
The priority is medium
Reviewed
Confirmed
Issue has been confirmed
Reviewed
Duplicate
This issue or pull request already exists
Reviewed
Invalid
Invalid issue
Reviewed
Won't Fix
This issue won't be fixed
Status
Abandoned
Somebody has started to work on this but abandoned work
Status
Blocked
Something is blocking this issue or pull request
Status
Need More Info
Feedback is required to reproduce issue or to continue work
No Label
Kind/Documentation
Kind/Feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
alex.sander i.galabov laurents linus maxsc micgor32 nedas sfisches siemer thomaswu tobiasd vincent weast
No Assignees linus micgor32
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: TEDective/etl#3
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?