Data World 2 · Pipeline Plumbing · Stage 4

Only the New

A bookmark remembers which files a job read; it cannot see a file rewritten in place or a row changed in a database: write so that reading twice is harmless.

  • Job bookmarks
  • transformation_ctx and job.commit
  • Idempotent writes
  • JDBC bookmark keys
Domain
Data engineering
World
Data World 2: Pipeline Plumbing (Associate)
Game type
Lake tuner: Tune a lake's tables until the queries fit their limit and budget.
Time
About 25 minutes on a first attempt (estimated)
Exam guides
  • DEA-C01Skill 1.1.11: Describe replayability of data ingestion pipelines; Skill 1.1.12: Define stateful and stateless data transactions; Skill 1.2.7: Troubleshoot and debug common transformation failures and performance issues; Skill 3.3.6: Troubleshoot and maintain pipelines (for example, AWS Glue, Amazon EMR)