{"id":68,"date":"2023-11-06T16:02:55","date_gmt":"2023-11-06T16:02:55","guid":{"rendered":"https:\/\/blog.dataplatform.lt\/?p=68"},"modified":"2023-11-06T16:02:55","modified_gmt":"2023-11-06T16:02:55","slug":"data-from-apis","status":"publish","type":"post","link":"https:\/\/blog.dataplatform.lt\/?p=68","title":{"rendered":"Data from APIs"},"content":{"rendered":"\n<p>Sometimes data for web page is loaded dynamically, i.e. on demand. This can be done for performance reasons, to make initial load faster. Sometimes you can identify this pattern by observing spinners or loaders, or content appearing incrementally:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/lh7-us.googleusercontent.com\/746VqhBqpcRzp6hxnrBtVE6PE8Kf0GQgadh2H3BacaeUISHODjdIDpx4rl2AtGmhbSWHPV1d5ze5jSroe0b30beOnhgoKiFDqdSyt-32gsop2XkiD7QD8NPJVzlPAGfnBfV_SMDcpt8B20PiBSfO7Jk\" width=\"287\" height=\"92\"><\/td><td><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/lh7-us.googleusercontent.com\/85BXrDsDukZ--3wQfRqHaAxX8lfRc6HyxNk68xBimPoFxPhCT45zbb86v-qjKqPq4R1bZAyxhhyIa6lRROoM3ZrYzjep0AVx-_t6gfBZYGZwesTBMKP3Y7SjQF7REwj0_c1IdhdeDQ88c9HS6EzXT9M\" width=\"286\" height=\"117\"><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>On the browser, this can be observed by inspecting <strong>Network<\/strong> tab of debugger tools. For example, on Chrome, you can find Network tab by clicking on menu \u201c<em>View > Developer > Developer Tools<\/em>\u201d and going into Network tab<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Network activity record\" width=\"840\" height=\"473\" src=\"https:\/\/www.youtube.com\/embed\/5LSncfnaSAo?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen><\/iframe>\n<\/div><figcaption class=\"wp-element-caption\">.<a href=\"https:\/\/www.youtube.com\/watch?v=5LSncfnaSAo\"> Watch screencast<\/a><\/figcaption><\/figure>\n\n\n\n<p>Having problems finding the right request? Try searching by data<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Search Network Request by data\" width=\"840\" height=\"473\" src=\"https:\/\/www.youtube.com\/embed\/Z-MAiXalmdI?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<p class=\"has-text-align-center\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/lh7-us.googleusercontent.com\/ikEVupeJX1HkrqYfMhMvpe1RAa0Kcn46Pp2BUgN6LdLs3Z9xsBBpNx-NUuIR8OJrsf8yi3oKJTIuj9LMAbVIp6JSbZfpbyZSNA55ywa3sBaDe3RYSsSGuAvTq8iN_fc-hlKOp6qheGfNHipAMBKKX80\" width=\"587\" height=\"312\"><\/p>\n\n\n\n<p>Now you can find API request which yields required data<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl https:\/\/gedimino37.lt\/\/catalog.php<\/code><\/pre>\n\n\n\n<p>Which would (in this case) would yield structured data:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;{\"id\":\"B0.1\",\"status\":\"sold\",\"direction\":\"PR\",\"floor\":\"0\",\"size\":\"87,69\",\"rooms\":\"6}]<\/code><\/pre>\n\n\n\n<p>The code snippet in Python to retrieve this kind of data:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import json\nfrom dphelper import DPHelper\n\nhelper = DPHelper()\nheaders = helper.create_headers(authority=\"gedimino37.lt\")\n\ncontent = helper.from_url('https:\/\/gedimino37.lt\/\/catalog.php', headers=headers)\ndata = json.loads(content)\n<\/code><\/pre>\n\n\n\n<p>Or, without or helper library (using standard requests library):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import requests\nfrom dphelper import DPHelper\n\nhelper = DPHelper()\nheaders = helper.create_headers(authority=\"gedimino37.lt\")\n\nr = requests.get('https:\/\/gedimino37.lt\/catalog.php', headers=headers)\ndata = r.json()<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Sometimes data for web page is loaded dynamically, i.e. on demand. This can be done for performance reasons, to make initial load faster. Sometimes you can identify this pattern by observing spinners or loaders, or content appearing incrementally: On the browser, this can be observed by inspecting Network tab of debugger tools. For example, on &hellip; <a href=\"https:\/\/blog.dataplatform.lt\/?p=68\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Data from APIs&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[],"class_list":["post-68","post","type-post","status-publish","format-standard","hentry","category-patterns-and-best-practises"],"_links":{"self":[{"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/posts\/68","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=68"}],"version-history":[{"count":1,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/posts\/68\/revisions"}],"predecessor-version":[{"id":69,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=\/wp\/v2\/posts\/68\/revisions\/69"}],"wp:attachment":[{"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=68"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=68"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.dataplatform.lt\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=68"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}