{"id":47,"date":"2015-09-04T09:28:06","date_gmt":"2015-09-04T09:28:06","guid":{"rendered":"http:\/\/daplab.ch\/?p=47"},"modified":"2015-10-07T12:27:23","modified_gmt":"2015-10-07T12:27:23","slug":"hdfs-hello-world","status":"publish","type":"post","link":"https:\/\/daplab.ch\/2015\/09\/04\/hdfs-hello-world\/","title":{"rendered":"HDFS Hello World"},"content":{"rendered":"<p>This page aims at creating a &#8220;copy-paste&#8221;-like tutorial to familiarize with <a href=\"http:\/\/hadoop.apache.org\/docs\/current\/hadoop-project-dist\/hadoop-hdfs\/HDFSCommands.html\">HDFS commands<\/a> . It mainly focuses on user commands (uploading and downloading data into HDFS).<\/p>\n<h1>Requirements<\/h1>\n<ul>\n<li>SSH (for Windows, use <a href=\"http:\/\/www.chiark.greenend.org.uk\/~sgtatham\/putty\/download.html\" target=\"_blank\">PuTTY<\/a>\u00a0and see\u00a0<a href=\"https:\/\/www.digitalocean.com\/community\/tutorials\/how-to-use-ssh-keys-with-putty-on-digitalocean-droplets-windows-users\" target=\"_blank\">how to create a key with PuTTY<\/a>)<\/li>\n<li>An account in the\u00a0<a href=\"http:\/\/daplab.ch\/\" target=\"_blank\">DAPLAB<\/a>, and send your ssh public key to Benoit.<\/li>\n<li>A browser &#8212; well, if you can access this page, you should have met this requirement\u00a0\ud83d\ude42<\/li>\n<\/ul>\n<div>\n<h1>Resources<\/h1>\n<p>While the source of truth for HDFS commands is the code source, the documentation page describing the\u00a0<code>hdfs dfs<\/code>\u00a0commands is really useful:<\/p>\n<\/div>\n<div>\n<ul>\n<li><a href=\"http:\/\/hadoop.apache.org\/docs\/current\/hadoop-project-dist\/hadoop-common\/FileSystemShell.html\">http:\/\/hadoop.apache.org\/docs\/current\/hadoop-project-dist\/hadoop-common\/FileSystemShell.html<\/a><\/li>\n<\/ul>\n<\/div>\n<h1>Basic Manipulations<\/h1>\n<h3>Listing a folder<\/h3>\n<h4>Your home folder<\/h4>\n<div>\n<pre class=\"console\">$ hdfs dfs -ls\r\nFound 28 items\r\n...\r\n-rw-r--r--   3 bperroud daplab_user    6398990 2015-03-13 11:01 data.csv\r\n...\r\n^^^^^^^^^^   ^ ^^^^^^^^ ^^^^^^^^^^^    ^^^^^^^ ^^^^^^^^^^ ^^^^^ ^^^^^^^^\r\n         1   2        3           4          5          6     7        8\r\n<\/pre>\n<div>Columns, as numbered below, represent:<\/div>\n<ol>\n<li>Permissions, in\u00a0<a class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/File_system_permissions#Notation_of_traditional_Unix_permissions\" rel=\"nofollow\">unix-style<\/a>\u00a0syntax<\/li>\n<li>Replication factor (RF in short), default being 3 for a file. Directories have a RF of 0.<\/li>\n<li>Owner<\/li>\n<li>Group owning the file<\/li>\n<li>Size of the file, in bytes. Note that to compute the physical space used, this number should be multiplied by the RF.<\/li>\n<li>Modification date. As HDFS is mostly a\u00a0<i><a class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Write_once_read_many\" rel=\"nofollow\">write-once-read-many<\/a><\/i>\u00a0filesystem, this date often means creation date<\/li>\n<li>Modification time. Same as date.<\/li>\n<li>Filename, within the listed folder<\/li>\n<\/ol>\n<\/div>\n<h4>Listing the \/tmp folder<\/h4>\n<div>\n<pre>$ hdfs dfs -ls \/tmp<\/pre>\n<\/div>\n<h3>Uploading a file<\/h3>\n<h4>In \/tmp<\/h4>\n<div>\n<pre class=\"console\">$ hdfs dfs -copyFromLocal localfile.txt \/tmp\/<\/pre>\n<\/div>\n<div>\n<div>The first arguments after\u00a0<code>-copyFromLocal<\/code>\u00a0point to local files or folders, while the last argument is a file (if only one file listed as source) or directory in HDFS.<\/div>\n<div>Note:\u00a0<code>hdfs dfs -put<\/code>\u00a0is doing about the same thing, but\u00a0<code>-copyFromLocal<\/code>\u00a0is more explicit when you&#8217;re uploading a local file and thus preferred.<\/div>\n<\/div>\n<div><\/div>\n<h3>Downloading a file<\/h3>\n<h4>From \/tmp<\/h4>\n<div>\n<pre class=\"console\">$ hdfs dfs -copyToLocal \/tmp\/remotefile.txt .<\/pre>\n<\/div>\n<div>\n<div>The first arguments after\u00a0<code class=\"console\">-copyToLocal<\/code>\u00a0point to files or folder in HDFS, while the last argument is a local file (if only one file listed as source) or directory.<\/div>\n<div><code>hdfs dfs -get<\/code>\u00a0is doing about the same thing, but\u00a0<code>-copyToLocal<\/code>\u00a0is more explicit when you&#8217;re downloading a file and thus preferred.<\/div>\n<div><\/div>\n<h3>Creating a folder<\/h3>\n<div>\n<h4>In your home folder<\/h4>\n<\/div>\n<pre class=\"console\">$ hdfs dfs -mkdir dummy-folder<\/pre>\n<div>\n<h4>In \/tmp<\/h4>\n<\/div>\n<pre class=\"console\">$ hdfs dfs -mkdir \/tmp\/dummy-folder<\/pre>\n<div>Note that relative paths points to your home folder,\u00a0<code>\/user\/bperroud<\/code>\u00a0for instance.<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>This page aims at creating a &#8220;copy-paste&#8221;-like tutorial to familiarize with HDFS commands . It mainly focuses on user commands (uploading and downloading data into HDFS). Requirements SSH (for Windows, use PuTTY\u00a0and see\u00a0how to create a key with PuTTY) An account in the\u00a0DAPLAB, and send your ssh public key to Benoit. A browser &#8212; well, [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[13],"tags":[7,6],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v15.6.2 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>HDFS Hello World - DAPLAB - Data Analysis and Processing Lab<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/daplab.ch\/2015\/09\/04\/hdfs-hello-world\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"HDFS Hello World - DAPLAB - Data Analysis and Processing Lab\" \/>\n<meta property=\"og:description\" content=\"This page aims at creating a &#8220;copy-paste&#8221;-like tutorial to familiarize with HDFS commands . It mainly focuses on user commands (uploading and downloading data into HDFS). Requirements SSH (for Windows, use PuTTY\u00a0and see\u00a0how to create a key with PuTTY) An account in the\u00a0DAPLAB, and send your ssh public key to Benoit. A browser &#8212; well, [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"http:\/\/daplab.ch\/2015\/09\/04\/hdfs-hello-world\/\" \/>\n<meta property=\"og:site_name\" content=\"DAPLAB - Data Analysis and Processing Lab\" \/>\n<meta property=\"article:published_time\" content=\"2015-09-04T09:28:06+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2015-10-07T12:27:23+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\">\n\t<meta name=\"twitter:data1\" content=\"2 minutes\">\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/daplab.ch\/#website\",\"url\":\"https:\/\/daplab.ch\/\",\"name\":\"DAPLAB - Data Analysis and Processing Lab\",\"description\":\"Reduces the entry barrier for companies to find value out of their data and ultimately turn into a data-driven company\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/daplab.ch\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"http:\/\/daplab.ch\/2015\/09\/04\/hdfs-hello-world\/#webpage\",\"url\":\"http:\/\/daplab.ch\/2015\/09\/04\/hdfs-hello-world\/\",\"name\":\"HDFS Hello World - DAPLAB - Data Analysis and Processing Lab\",\"isPartOf\":{\"@id\":\"https:\/\/daplab.ch\/#website\"},\"datePublished\":\"2015-09-04T09:28:06+00:00\",\"dateModified\":\"2015-10-07T12:27:23+00:00\",\"author\":{\"@id\":\"https:\/\/daplab.ch\/#\/schema\/person\/a2e383674c25ef6233dd15f3d269c7c0\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/daplab.ch\/2015\/09\/04\/hdfs-hello-world\/\"]}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/daplab.ch\/#\/schema\/person\/a2e383674c25ef6233dd15f3d269c7c0\",\"name\":\"Daplab\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/daplab.ch\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/dc16fc8fae1e733c6bef9181443db346?s=96&d=mm&r=g\",\"caption\":\"Daplab\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/daplab.ch\/wp-json\/wp\/v2\/posts\/47"}],"collection":[{"href":"https:\/\/daplab.ch\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/daplab.ch\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/daplab.ch\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/daplab.ch\/wp-json\/wp\/v2\/comments?post=47"}],"version-history":[{"count":2,"href":"https:\/\/daplab.ch\/wp-json\/wp\/v2\/posts\/47\/revisions"}],"predecessor-version":[{"id":50,"href":"https:\/\/daplab.ch\/wp-json\/wp\/v2\/posts\/47\/revisions\/50"}],"wp:attachment":[{"href":"https:\/\/daplab.ch\/wp-json\/wp\/v2\/media?parent=47"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/daplab.ch\/wp-json\/wp\/v2\/categories?post=47"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/daplab.ch\/wp-json\/wp\/v2\/tags?post=47"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}