Vagrant project to spin up a cluster of 4 virtual machines with Hadoop v2.6.0 and Spark v1.3.0.
Trusted Windows (PC) download Spark Broadband Assist 1.0.1.92. Virus-free and 100% clean download. Get Spark Broadband Assist alternative downloads. Spark your ideas.Inspiration is an important part of process. Spark is made of two decks of inspirational prompts based on the dichotomy of Discipline & Impulse. Just tap a deck.
Some gotcha's.
If you have the resources (CPU + Disk Space + Memory), you may modify Vagrantfile to have even more HDFS DataNodes, YARN NodeManagers, and Spark slaves. Just find the line that says 'numNodes = 4' in Vagrantfile and increase that number. The scripts should dynamically provision the additional slaves for you.
You can make the VM setup even faster if you pre-download the Hadoop, Spark, and Oracle JDK into the /resources directory.
The setup script will automatically detect if these files (with precisely the same names) exist and use them instead. If you are using slightly different versions, you will have to modify the script accordingly.
I typically run the following tests after post-provisioning on node1 (as root user).
Test YARN
Run the following command to make sure you can run a MapReduce job.
Test Spark on YARN
You can test if Spark can run on YARN by issuing the following command. Try NOT to run this command on the slave nodes.
Test code directly on SparkTest Spark using Shell
Start the Spark shell using the following command. Try NOT to run this command on the slave nodes.
Then go here https://spark.apache.org/docs/latest/quick-start.html to start the tutorial. Most likely, you will have to load data into HDFS to make the tutorial work (Spark cannot read data on the local file system).
You might also want to dive into the learn-scala folder as that is a companion Scala project to learn Spark.
You can check the following URLs to monitor the Hadoop daemons.
A list of available Vagrant boxes is shown at http://www.vagrantbox.es.
The Vagrant box is downloaded to the ~/.vagrant.d/boxes directory. On Windows, this is C:/Users/{your-username}/.vagrant.d/boxes.
This project was kludge together with great pointers from all around the internet. All references made inside the files themselves.
Copyright 2014 Jee Vang
Licensed under the Apache License, Version 2.0 (the 'License');you may not use this file except in compliance with the License.You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an 'AS IS' BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.
Comments are closed.
|
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. Archives
January 2023
Categories |