{"id":131,"date":"2024-04-19T08:24:31","date_gmt":"2024-04-19T08:24:31","guid":{"rendered":"https:\/\/blog.dataplatform.lt\/?p=131"},"modified":"2024-04-19T08:24:31","modified_gmt":"2024-04-19T08:24:31","slug":"enforcing-data-freshness","status":"publish","type":"post","link":"https:\/\/blog.dataplatform.lt\/?p=131","title":{"rendered":"Enforcing data freshness"},"content":{"rendered":"\n<p>Sometimes we reuse output from other pipelines using this pattern:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>helper = DPHelper(is_verbose=True) \nr = helper.snapshot.get_latest(by_challenge_id=140) <\/code><\/pre>\n\n\n\n<p>We do have staleness checks on the pipelines. However, if pipeline reuses other pipeline data the pattern above would silently fail. <\/p>\n\n\n\n<p>To avoid that situation, since dphelper v0.16 version we added optional parameter <strong>max_stale_days<\/strong>.<\/p>\n\n\n\n<p>First, make sure you update your pip package localy.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pip3 install dphelper -U<\/code><\/pre>\n\n\n\n<p>Check your package version <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pip show dphelper<\/code><\/pre>\n\n\n\n<p>If everything OK, then use this pattern<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>helper = DPHelper(is_verbose=True) \nr = helper.snapshot.get_latest(by_challenge_id=140, max_stale_days=14) <\/code><\/pre>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Sometimes we reuse output from other pipelines using this pattern: We do have staleness checks on the pipelines. However, if pipeline reuses other pipeline data the pattern above would silently fail. To avoid that situation, since dphelper v0.16 version we added optional parameter max_stale_days. First, make sure you update your pip package localy. Check your &hellip; <a href=\"https:\/\/blog.dataplatform.lt\/?p=131\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Enforcing data freshness&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[],"class_list":["post-131","post","type-post","status-publish","format-standard","hentry","category-patterns-and-best-practises"],"_links":{"self":[{"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/posts\/131","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=131"}],"version-history":[{"count":1,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/posts\/131\/revisions"}],"predecessor-version":[{"id":132,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/posts\/131\/revisions\/132"}],"wp:attachment":[{"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=131"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=131"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=131"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}