{"id":135,"date":"2024-05-17T09:41:07","date_gmt":"2024-05-17T09:41:07","guid":{"rendered":"https:\/\/blog.dataplatform.lt\/?p=135"},"modified":"2024-05-17T09:41:07","modified_gmt":"2024-05-17T09:41:07","slug":"streaming-json","status":"publish","type":"post","link":"https:\/\/blog.dataplatform.lt\/?p=135","title":{"rendered":"Streaming JSON"},"content":{"rendered":"\n<p>Sometimes data stored in memory can become very large, for example the pattern of pagination here:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># ssva.lt energinio naudingumo registras\n\n# from pprint import pprint\nimport re\nimport json\nfrom dphelper import DPHelper\n\nhelper = DPHelper(is_verbose=True)\nheaders = helper.create_headers(authority='ssva.lt')\n\ndef generate_url(page_number, PAGE_SIZE=10000):\n    return f'https:\/\/www.ssva.lt\/registrai\/pensreg\/pensert_list.php?goto={page_number}&amp;pagesize={PAGE_SIZE}'\n\nPAGINATION_DATA_PATTERN = ('Toliau&lt;\/a>&lt;\/li>.*?pageNum=\"(.*?)\" >Pabaiga&lt;\/a>')\n\nDATA_PATTERN = (\n    '&lt;tr  id=\"gridRow.*?class=\"r-gridrow\">'\n    '&lt;tddata-record-id=\".*?\".*?SertifikatoNr\".*?val=\"(.*?)\" >' # sert nr\n    '.*?\"IsdavimoD\".*?val=\"(.*?)\" >' # isdavimo data\n    '.*?GaliojimoD\".*?val=\"(.*?)\" >' # galiojimo data\n    '.*?UnikalusNr\".*?val=\"(.*?)\" >' # unikalus nr\n    '.*?Adresas\".*?val=\"(.*?)\" >' # adresas\n    '.*?Paskirtis.*?val=\"(.*?)\" >' # paskirtis\n    '.*?PEN\".*?val=\"(.*?)\" >' # PEN\n    '.*?Ap\" >.*?val=\"(.*?)\" >' # sildomas plotas\n    '.*?Q\".*?val=\"(.*?)\" >' # E.sanaudos\n    '.*?Hsrc\".*?val=\"(.*?)\" >' # silumos saltinis\n    '.*?Pastaba\".*?val=\"(.*?)\" >&lt;\/span>&lt;\/span>&lt;\/td>&lt;\/tr>' # pastaba\n)\n\ndef get_page_count():\n    FIRST_PAGE_URL = generate_url(1)\n    raw_content = helper.from_url(FIRST_PAGE_URL, headers=headers)\n    rg = re.compile(PAGINATION_DATA_PATTERN)\n    last_page_nr = rg.findall(raw_content)\n    return int(last_page_nr&#91;0])\n\nif __name__ == \"__main__\":\n    last_page_nr = get_page_count()\n    # Compile is slow, so we do it once\n    rg = re.compile(DATA_PATTERN)\n\n    data = &#91;]\n    for page_number in range(1, last_page_nr + 1):\n        url_for_read = generate_url(page_number)\n        # print(url_for_read)\n        page_content = helper.from_url(url_for_read, headers=headers)\n        results = rg.findall(page_content)\n        data.extend(results)\n        print(json.dumps(results, indent=2))\n\n    print(json.dumps(data, indent=2))\n <\/code><\/pre>\n\n\n\n<p>In such scenarios it can be useful to define this as streamin g response from <a href=\"https:\/\/pypi.org\/project\/json-stream\/\">json_stream pip package<\/a>. Basic example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import sys\nimport json\n\nfrom json_stream import streamable_list\n\ndef test_stream():\n    for i in range(20):\n        yield i\n \n\n# wrap existing iterable\ndata = streamable_list(test_stream())\n\n# consume iterable with standard json.dump()\njson.dump(data, sys.stdout)<\/code><\/pre>\n\n\n\n<p>Applying example to this scenario:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># ssva.lt energinio naudingumo registras\n\n# from pprint import pprint\nimport re\nimport json\nimport sys\nfrom dphelper import DPHelper\n\n\nfrom json_stream import streamable_list\n\nhelper = DPHelper(is_verbose=True)\nheaders = helper.create_headers(authority='ssva.lt')\n\ndef generate_url(page_number, PAGE_SIZE=100):\n    return f'https:\/\/www.ssva.lt\/registrai\/pensreg\/pensert_list.php?goto={page_number}&amp;pagesize={PAGE_SIZE}'\n\nPAGINATION_DATA_PATTERN = ('Toliau&lt;\/a>&lt;\/li>.*?pageNum=\"(.*?)\" >Pabaiga&lt;\/a>')\n\nDATA_PATTERN = (\n    '&lt;tr  id=\"gridRow.*?class=\"r-gridrow\">'\n    '&lt;tddata-record-id=\".*?\".*?SertifikatoNr\".*?val=\"(.*?)\" >' # sert nr\n    '.*?\"IsdavimoD\".*?val=\"(.*?)\" >' # isdavimo data\n    '.*?GaliojimoD\".*?val=\"(.*?)\" >' # galiojimo data\n    '.*?UnikalusNr\".*?val=\"(.*?)\" >' # unikalus nr\n    '.*?Adresas\".*?val=\"(.*?)\" >' # adresas\n    '.*?Paskirtis.*?val=\"(.*?)\" >' # paskirtis\n    '.*?PEN\".*?val=\"(.*?)\" >' # PEN\n    '.*?Ap\" >.*?val=\"(.*?)\" >' # sildomas plotas\n    '.*?Q\".*?val=\"(.*?)\" >' # E.sanaudos\n    '.*?Hsrc\".*?val=\"(.*?)\" >' # silumos saltinis\n    '.*?Pastaba\".*?val=\"(.*?)\" >&lt;\/span>&lt;\/span>&lt;\/td>&lt;\/tr>' # pastaba\n)\n\ndef get_page_count():\n    FIRST_PAGE_URL = generate_url(1)\n    raw_content = helper.from_url(FIRST_PAGE_URL, headers=headers)\n    rg = re.compile(PAGINATION_DATA_PATTERN)\n    last_page_nr = rg.findall(raw_content)\n    return int(last_page_nr&#91;0])\n\ndef yield_data():\n    last_page_nr = get_page_count()\n    rg = re.compile(DATA_PATTERN)\n\n    last_page_nr = 5\n\n    for page_number in range(1, last_page_nr + 1):\n        url_for_read = generate_url(page_number)\n        page_content = helper.from_url(url_for_read, headers=headers)\n        results = rg.findall(page_content)\n\n        for result in results:\n            yield result\n       \n\nif __name__ == \"__main__\":\n    # wrap existing iterable\n    data = streamable_list(yield_data())\n\n    # consume iterable with standard json.dump()\n    json.dump(data, sys.stdout)\n\n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Sometimes data stored in memory can become very large, for example the pattern of pagination here: In such scenarios it can be useful to define this as streamin g response from json_stream pip package. Basic example: Applying example to this scenario:<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[],"class_list":["post-135","post","type-post","status-publish","format-standard","hentry","category-patterns-and-best-practises"],"_links":{"self":[{"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/posts\/135","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=135"}],"version-history":[{"count":1,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/posts\/135\/revisions"}],"predecessor-version":[{"id":137,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/posts\/135\/revisions\/137"}],"wp:attachment":[{"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=135"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=135"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=135"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}